The DOI matches its DataCite record; 9 of the 9 references it cites were found in a registry or an index.
Distilling the Knowledge in a Neural Network
Open accessAbstract
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“According to [19], both formulations should be equivalent in the limit of a high smoothing factor for logits (cf. § 3.2), but our choice of not smoothing leads to this different behavior.”
From Incremental Learning of Object Detectors without Catastrophic Forgetting (2017) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These prior methods are highly effective on individual tasks, but they are not few-shot learners, require privileged access to datasets of human demonstrations or a target policy, and struggle to generalise to held-out tasks 25 .”
From Learning few-shot imitation as cultural transmission (2023) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The key insight lies in the hypothesis that LLMs contain hidden knowledge (Hinton et al., 2015), latent representations within their weights that are not fully accessible through standard inference.”
From TT-SI: Self-Improving LLM Agents with Test-Time Training (2026) · cites this paper as Hinton et al., 2015
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“KD technology originated from the concept of transferring knowledge from large models to small models, and was formally proposed by (Hinton et al. 2015) in 2015.”
From Survey: federated learning data security and privacy-preserving in edge-Internet of Things (2024) · cites this paper as Hinton et al. 2015
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These challenges have motivated researchers to propose techniques for reducing transformers' size, latency and energy consumption for efficient inference for a wide range of applications.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [45]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“During training, finetuning a model on step-by-step reasoning traces generated by a teacher model can transfer much of the latter's reasoning capability, substantially outperforming fine-tuning on final answers alone (Hinton et al., 2015;Gou et al., 2021;Shridhar et al., 2023;Chen et al., 2025b).”
From How to Steal Reasoning Without Reasoning Traces (2026) · cites this paper as Hinton et al., 2015
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As noted in [19], knowledge distillation can also be expressed as a cross-entropy loss.”
From Incremental Learning of Object Detectors without Catastrophic Forgetting (2017) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“We distill logits without any smoothing, unlike [19], because most of the proposals already produce a smooth distribution of scores. Moreover, in our case, both the old and the new networks are similar with almost the same parameters (in the beginning), and so smoothing the logits distribution is not necessary to stabilize the learning.”
From Incremental Learning of Object Detectors without Catastrophic Forgetting (2017) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“Knowledge distillation, introduced by Hinton et al. [1], facilitates knowledge transfer from a large, cumbersome 'teacher' model to a smaller and more compact 'student' model.”
From Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Knowledge distillation provides a degree of controllable compression by allowing us to transfer knowledge from a larger model to a smaller one.”
From Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Our approach provides a clear advantage by enabling a greater compression of the teacher model compared to the previous methods that relied on pruning [10-15], knowledge distillation (KD) [1,3,5,7,8], and the combination of both [16-20].”
From Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Knowledge distillation is another approach that involves training a smaller “student” model to replicate the behavior of a larger “teacher” model, requiring less computational power while retaining performance [161].”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [161]
The cited source says what the claim attributes to it.
In the citing paper
“In the context of action recognition (Tran et al. 2015; Wang et al. 2016; Carreira and Zisserman 2017; Feichtenhofer et al. 2019), there are handful of studies towards this end (Hoffman, Gupta, and Darrell 2016; Luo et al. 2018; Garcia, Morerio, and Murino 2018, 2019; Garcia et al. 2019; Stroud et al. 2020), all of which follow transfer learning scheme via knowledge distillation (Hinton et al. 201”
From Towards Good Practices for Missing Modality Robust Action Recognition (2023) · cites this paper as Hinton et al. 2015
The cited source says what the claim attributes to it.
In the citing paper
“Fine-Tuning (FT) based methods form a group which often uses a distillation [27] term to reduce catastrophic forgetting [44].”
From A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Feature maps [8, 9, 10] and logits of a network [11, 12] are widely used as knowledge.”
From PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation (2021) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“Temperature scaling is commonly used in settings such as knowledge distillation (Hinton et al., 2015) and statistical mechanics (Jaynes, 1957).”
From On calibration of modern neural networks (2017) · cites this paper as Hinton et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“Training on q(y | x; θ T ) is attractive since it gives more information about other classes for a given data point (e.g.similarity between classes) and has less variance in gradients (Hinton et al., 2015).”
From Sequence-Level Knowledge Distillation (2016) · cites this paper as Hinton et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“In this paper we consider applying knowledge distillation approaches (Bucila et al., 2006;Hinton et al., 2015) that have proven successful for reducing the size of neural models in other domains to the problem of NMT.”
From Sequence-Level Knowledge Distillation (2016) · cites this paper as Hinton et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“Such a training improves the performance of the student model (alternatively, it can be regarded as a form of compression of the teacher model into a smaller one - the student).”
From Training data-efficient image transformers & distillation through attention (2020) · cites this paper as [24]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Distillation turns out to be a more effective method for obtaining interpretable models.”
From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [102]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our improvement compared to the implementation of [15] can be explained by the use of a larger projection head (3-layer, use of batchnormalizations and projection dimension of 256 ).”
From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [15]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We obtain 71 . 1% for MoCo after 300 epochs of training while [15] report 71 . 1% after 800 epochs of training.”
From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [15]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Self-citation(2014) · Cited by 33,419
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationEnsemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Self-citationAdaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384
Self-citationLarge Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,202
Self-citationBuciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068
Learning small-size DNN with output-distribution-based criteria
Li, J., Zhao, R., Huang, J.-T., & Gong, Y. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-432 · Cited by 252
Cited by
Sorted by citations
Items 201–300 of 342
Principles and Practice of Explainable Machine Learning
Belle, V., & Papantonis, I. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.688969 · Cited by 99
Survey on Explainable AI: Techniques, challenges and open issues
Abusitta, A., Li, M. Q., & Fung, B. C. M. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124710 · Cited by 98
An End-to-End Architecture for Class-Incremental Object Detection with Knowledge Distillation
Hao, Y., Fu, Y., Jiang, Y.-G., & Tian, Q. (2019) · 2019 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2019.00009 · Cited by 98
Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution
Chu, L., Hu, X., Hu, J., Wang, L., & Pei, J. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220063 · Cited by 87
From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough
Mars, M. (2022) · Applied Sciences · doi:10.3390/app12178805 · Cited by 91
Structured Pruning of Neural Networks With Budget-Aware Regularization
Lemaire, C., Achkar, A., & Jodoin, P.-M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00932 · Cited by 90
Learning high-accuracy error decoding for quantum processors
Bausch, J., Senior, A. W., Heras, F. J. H., Edlich, T., Davies, A., Newman, M., Jones, C., Satzinger, K., Niu, M. Y., Blackwell, S., et al. (2024) · Nature · doi:10.1038/s41586-024-08148-8 · Cited by 106
“Secondly, techniques such as distillation [45], lower precision inference and weight pruning can be applied to achieve similar performance with less computation.”
Federated Contrastive Learning With Feature-Based Distillation for Human Activity Recognition
Xiao, Z., & Tong, H. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3510428 · Cited by 86
Federated Learning for Privacy-Preserving Open Innovation Future on Digital Health
Long, G., Shen, T., Tan, Y., Gerrard, L., Clarke, A., & Jiang, J. (2022) · Humanity Driven AI · doi:10.1007/978-3-030-72188-6_6 · Cited by 85
A Compact Convolutional Neural Network for Surface Defect Inspection
Huang, Y., Qiu, C., Wang, X., Wang, S., & Yuan, K. (2020) · Sensors · doi:10.3390/s20071974 · Cited by 85
Overcoming Multi-Model Forgetting in One-Shot NAS With Diversity Maximization
Zhang, M., Li, H., Pan, S., Chang, X., & Su, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00783 · Cited by 83
Zhou, H., Du, J., Zhang, Y., Wang, Q., Liu, Q.-F., & Lee, C.-H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3096037 · Cited by 85
PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation
Kim, J., Chang, S., & Kwak, N. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-248 · Cited by 85
“Feature maps [8, 9, 10] and logits of a network [11, 12] are widely used as knowledge.”
BackgroundModel Evasion Attack on Intrusion Detection Systems using Adversarial Machine Learning
Ayub, M. A., Johnson, W. A., Talbert, D. A., & Siraj, A. (2020) · 2020 54th Annual Conference on Information Sciences and Systems (CISS) · doi:10.1109/ciss48834.2020.1570617116 · Cited by 86
Distilling Reasoning Capabilities into Smaller Language Models
Shridhar, K., Stolfo, A., & Sachan, M. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.441 · Cited by 79
“Knowledge Distillation Our approach is reminiscent of knowledge distillation (Ba and Caru-ana, 2014; Hinton et al., 2015) in that we use a student network to mimic the large teacher language model.”
BackgroundMethodInterruption-Aware Cooperative Perception for V2X Communication-Aided Autonomous Driving
Ren, S., Lei, Z., Wang, Z., Dianati, M., Wang, Y., Chen, S., & Zhang, W. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3371974 · Cited by 75
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92
Imitation Attacks and Defenses for Black-box Machine Translation Systems
Wallace, E., Stern, M., & Song, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.446 · Cited by 75
Agentic Large Language Models, a Survey
Plaat, A., Van Duijn, M., Van Stein, N., Preuss, M., Van der Putten, P., & Batenburg, K. J. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.18675 · Cited by 75
Wireless Network Intelligence at the Edge
Park, J., Samarakoon, S., Bennis, M., & Debbah, M. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2941458 · Cited by 92
Chung, J., Kamar, E., & Amershi, S. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.34 · Cited by 72
NTIRE 2023 Challenge on Efficient Super-Resolution: Methods and Results
Li, Y., Zhang, Y., Timofte, R., Van Gool, L., Yu, L., Li, Y., Li, X., Jiang, T., Wu, Q., Han, M., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00189 · Cited by 72
Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space
Chavan, A., Shen, Z., Liu, Z., Liu, Z., Cheng, K.-T., & Xing, E. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00488 · Cited by 72
Survey: Leakage and Privacy at Inference Time
Jegorova, M., Kaul, C., Mayor, C., O'Neil, A. Q., Weir, A., Murray-Smith, R., & Tsaftaris, S. A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3229593 · Cited by 77
Data-free Knowledge Distillation for Object Detection
Chawla, A., Yin, H., Molchanov, P., & Alvarez, J. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00333 · Cited by 70
Protein design via deep learning
Ding, W., Nakai, K., & Gong, H. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac102 · Cited by 66
Trustworthy AI in the Age of Pervasive Computing and Big Data
Kumar, A., Braud, T., Tarkoma, S., & Hui, P. (2020) · 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops48775.2020.9156127 · Cited by 68
Analog/Mixed-Signal Hardware Error Modeling for Deep Learning Inference
Rekhi, A. S., Zimmer, B., Nedovic, N., Liu, N., Venkatesan, R., Wang, M., Khailany, B., Dally, W. J., & Gray, C. T. (2019) · Proceedings of the 56th Annual Design Automation Conference 2019 · doi:10.1145/3316781.3317770 · Cited by 67
MobileDA: Toward Edge-Domain Adaptation
Yang, J., Zou, H., Cao, S., Chen, Z., & Xie, L. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2976762 · Cited by 67
The cascading neural network: building the Internet of Smart Things
Leroux, S., Bohez, S., De Coninck, E., Verbelen, T., Vankeirsbilck, B., Simoens, P., & Dhoedt, B. (2017) · Knowledge and Information Systems · doi:10.1007/s10115-017-1029-1 · Cited by 59
Federated Learning Empowered Real-Time Medical Data Processing Method for Smart Healthcare
Guo, K., Chen, T., Ren, S., Li, N., Hu, M., & Kang, J. (2024) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3185395 · Cited by 63
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., & Metzler, D. (2021) · arXiv · doi:10.48550/arxiv.2109.10686 · Cited by 63
Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models
McDonald, J., Li, B., Frey, N., Tiwari, D., Gadepally, V., & Samsi, S. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.151 · Cited by 62
Binh, L. M., & Woo, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19886 · Cited by 63
Towards Good Practices for Missing Modality Robust Action Recognition
Woo, S., Lee, S., Park, Y., Nugroho, M. A., & Kim, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25378 · Cited by 67
Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime
Dai, D., & Van Gool, L. (2018) · ETH Zurich · doi:10.3929/ethz-b-000310974 · Cited by 60
Secure and robust machine learning for healthcare: A survey
Qayyum, A., Qadir, J., Bilal, M., & Al-Fuqaha, A. (2021) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.3013489 · Cited by 59
Liao, Z., Girgis, H., Abdi, A., Vaseli, H., Hetherington, J., Rohling, R., Gin, K., Tsang, T., & Abolmaesumi, P. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2959209 · Cited by 56
Kadam, K. D., Ahirrao, S., & Kotecha, K. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/6845326 · Cited by 59
RetractedEasiEdge: A Novel Global Deep Neural Networks Pruning Method for Efficient Edge Computing
Yu, F., Cui, L., Wang, P., Han, C., Huang, R., & Huang, X. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3034925 · Cited by 58
Protecting Intellectual Property of Language Generation APIs with Lexical Watermark
He, X., Xu, Q., Lyu, L., Wu, F., & Wang, C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21321 · Cited by 56
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2023) · arXiv · doi:10.48550/arxiv.2312.10549 · Cited by 54
Model-Heterogeneous Semi-Supervised Federated Learning for Medical Image Segmentation
Ma, Y., Wang, J., Yang, J., & Wang, L. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3348982 · Cited by 55
Discovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.09251 · Cited by 53
Yee, K., Tantipongpipat, U., & Mishra, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479594 · Cited by 59
Decoupling and Recoupling Spatiotemporal Representation for RGB-D-based Motion Recognition
Zhou, B., Wang, P., Wan, J., Liang, Y., Wang, F., Zhang, D., Lei, Z., Li, H., & Jin, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01952 · Cited by 53
Visual Scene Understanding for Autonomous Driving Using Semantic Segmentation
Hofmarcher, M., Unterthiner, T., Arjona-Medina, J., Klambauer, G., Hochreiter, S., & Nessler, B. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_15 · Cited by 51
Pseudo Labels for Unsupervised Domain Adaptation: A Review
Li, Y., Guo, L., & Ge, Y. (2023) · Electronics · doi:10.3390/electronics12153325 · Cited by 51
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2024) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2024.3966 · Cited by 54
EPNet: Learning to Exit with Flexible Multi-Branch Network
Dai, X., Kong, X., & Guo, T. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3411973 · Cited by 48
The False Promise of Imitating Proprietary LLMs
Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., & Song, D. (2023) · arXiv · doi:10.48550/arxiv.2305.15717 · Cited by 48
“In theory, this imitation process may provide an easy method to distill (Hinton et al., 2014) the capabilities of any proprietary model, thus implying that open-source LMs will always be competitive with their commercial counterparts.”
BackgroundMethodAdam, M., & Baroudi, U. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3506214 · Cited by 46
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
Liu, L., Ren, X., Shang, J., Gu, X., Peng, J., & Han, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1153 · Cited by 45
Enhancing Reliability of Analog Neural Network Processors
Moon, S., Shin, K., & Jeon, D. (2019) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2019.2893256 · Cited by 45
Knowledge from the Original Network: Restore a Better Pruned Network with Knowledge Distillation
Chen, L., Chen, Y., Xi, J., & Le, X. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-020-00248-y · Cited by 48
Residual error based knowledge distillation
Gao, M., Wang, Y., & Wan, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.10.113 · Cited by 45
Machine Learning Explanations to Prevent Overtrust in Fake News Detection
Mohseni, S., Yang, F., Pentyala, S., Du, M., Liu, Y., Lupfer, N., Hu, X., Ji, S., & Ragan, E. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18072 · Cited by 44
Self-supervised transfer learning of physiological representations from free-living wearable data
Spathis, D., Perez-Pozuelo, I., Brage, S., Wareham, N. J., & Mascolo, C. (2021) · Proceedings of the Conference on Health, Inference, and Learning · doi:10.1145/3450439.3451863 · Cited by 40
Sense and Learn: Self-supervision for omnipresent sensors
Saeed, A., Ungureanu, V., & Gfeller, B. (2021) · Machine Learning with Applications · doi:10.1016/j.mlwa.2021.100152 · Cited by 34
Continual Learning for Generative Retrieval over Dynamic Corpora
Chen, J., Zhang, R., Guo, J., de Rijke, M., Chen, W., Fan, Y., & Cheng, X. (2023) · Proceedings of the 32nd ACM International Conference on Information and Knowledge Management · doi:10.1145/3583780.3614821 · Cited by 40
A Novel Lightweight IoT Intrusion Detection Model Based on Self-Knowledge Distillation
Wang, Z., Zhou, R., Yang, S., He, D., & Chan, S. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3533092 · Cited by 38
Visual Semantic Information Pursuit: A Survey
Liu, D., Bober, M., & Kittler, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2950025 · Cited by 39
A Strategy for an Uncompromising Incremental Learner
Venkatesan, R., Venkateswara, H., Panchanathan, S., & Li, B. (2017) · arXiv · doi:10.48550/arxiv.1705.00744 · Cited by 37
MELTing Point: Mobile Evaluation of Language Transformers
Laskaridis, S., Katevas, K., Minto, L., & Haddadi, H. (2024) · Proceedings of the 30th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3636534.3690668 · Cited by 37
Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learning
Le, B. M., & Woo, S. S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02045 · Cited by 36
Dar, S. S., Rehman, M. Z. U., Bais, K., Haseeb, M. A., & Kumar, N. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125337 · Cited by 36
Wen, H., Wu, Y., Yang, C., Duan, H., & Yu, S. (2020) · IEEE INFOCOM 2020 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) · doi:10.1109/infocomwkshps50562.2020.9162672 · Cited by 41
Understanding the Dark Side of LLMs' Intrinsic Self-Correction
Zhang, Q., Wang, D., Qian, H., Li, Y., Zhang, T., Huang, M., Xu, K., Li, H., Yan, L., & Qiu, H. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1314 · Cited by 36
Compressed Deep Learning to Classify Arrhythmia in an Embedded Wearable Device
Lee, K.-S., Park, H.-J., Kim, J. E., Kim, H. J., Chon, S., Kim, S., Jang, J., Kim, J.-K., Jang, S., Gil, Y., et al. (2022) · Sensors · doi:10.3390/s22051776 · Cited by 36
Soro, S. (2021) · arXiv · doi:10.48550/arxiv.2102.01255 · Cited by 36
Continual Learning for Fake Audio Detection
Ma, H., Yi, J., Tao, J., Bai, Y., Tian, Z., & Wang, C. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-794 · Cited by 39
Attention Weighted Local Descriptors
Wang, C., Xu, R., Lu, K., Xu, S., Meng, W., Zhang, Y., Fan, B., & Zhang, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3266728 · Cited by 36
Enabling Edge Devices that Learn from Each Other: Cross Modal Training for Activity Recognition
Xing, T., Sandha, S. S., Balaji, B., Chakraborty, S., & Srivastava, M. (2018) · Proceedings of the 1st International Workshop on Edge Systems, Analytics and Networking · doi:10.1145/3213344.3213351 · Cited by 35
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.09390 · Cited by 34
IF-CNN: Image-Aware Inference Framework for CNN With the Collaboration of Mobile Devices and Cloud
Shu, G., Liu, W., Zheng, X., & Li, J. (2018) · IEEE Access · doi:10.1109/access.2018.2880196 · Cited by 30
A Generalization Performance Study Using Deep Learning Networks in Embedded Systems
Gorospe, J., Mulero, R., Arbelaitz, O., Muguerza, J., & Antón, M. Á. (2021) · Sensors · doi:10.3390/s21041031 · Cited by 32
A Unified Multimodal De- and Re-Coupling Framework for RGB-D Motion Recognition
Zhou, B., Wang, P., Wan, J., Liang, Y., & Wang, F. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3274783 · Cited by 38
From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
Geng, X., Wang, Z., Chen, C., Xu, Q., Xu, K., Jin, C., Gupta, M., Yang, X., Chen, Z., Sabry Aly, M. M., et al. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3394494 · Cited by 26
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications
Park, J., Samarakoon, S., Elgabli, A., Kim, J., Bennis, M., Kim, S.-L., & Debbah, M. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3055679 · Cited by 25
Fonseca, E., Hershey, S., Plakal, M., Ellis, D. P. W., Jansen, A., & Moore, R. C. (2020) · IEEE Signal Processing Letters · doi:10.1109/lsp.2020.3006378 · Cited by 26
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
Yin, Y., Chen, C., Shang, L., Jiang, X., Chen, X., & Liu, Q. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.400 · Cited by 24
Predicting self-intercepted medication ordering errors using machine learning
King, C. R., Abraham, J., Fritz, B. A., Cui, Z., Galanter, W., Chen, Y., & Kannampallil, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254358 · Cited by 22
Alahmari, S., & Alghamdi, I. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.107849 · Cited by 21
Wang, B., Tang, Y., Wei, F., Ba, Z., & Ren, K. (2024) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2024.3492796 · Cited by 20
Ghani, B., Kalkman, V. J., Planqué, B., Vellinga, W.-P., Gill, L., & Stowell, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-00996-2 · Cited by 22
Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning
Malihi, L., & Heidemann, G. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7030154 · Cited by 20
A unified privacy preserving model with AI at the edge for human-in-the-loop cyber-physical systems
Rivadeneira, J. E., Borges, G. A., Rodrigues, A., Boavida, F., & Sá Silva, J. (2024) · Internet of Things · doi:10.1016/j.iot.2023.101034 · Cited by 18
Roy, K., Gaur, M., Soltani, M., Rawte, V., Kalyan, A., & Sheth, A. (2023) · Frontiers in Big Data · doi:10.3389/fdata.2022.1056728 · Cited by 18
SemGloVe: Semantic Co-Occurrences for GloVe From BERT
Gan, L., Teng, Z., Zhang, Y., Zhu, L., Wu, F., & Yang, Y. (2022) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2022.3197316 · Cited by 22
Benchmarking TensorFlow Lite Quantization Algorithms for Deep Neural Networks
Orasan, I. L., Seiculescu, C., & Caleanu, C. D. (2022) · 2022 IEEE 16th International Symposium on Applied Computational Intelligence and Informatics (SACI) · doi:10.1109/saci55618.2022.9919465 · Cited by 18
Li, S., Liu, F., Peng, G., Cheng, F., Zhao, B., & Ji, M. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3214300 · Cited by 18
Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
Igl, M., Farquhar, G., Luketina, J., Boehmer, W., & Whiteson, S. (2020) · arXiv · doi:10.48550/arxiv.2006.05826 · Cited by 17
Cloud-Edge Collaborative Defect Detection Based on Efficient Yolo Networks and Incremental Learning
Lei, Z., Zhang, Y., Wang, J., & Zhou, M. (2024) · Sensors · doi:10.3390/s24185921 · Cited by 16
From bias to black boxes: understanding and managing the risks of AI – an actuarial perspective
du Preez, V., Bennet, S., Byrne, M., Couloumy, A., Das, A., Dessain, J., Galbraith, R., King, P., Mutanga, V., Schiller, F., et al. (2024) · British Actuarial Journal · doi:10.1017/s1357321724000060 · Cited by 15
A comparative study on the impact of model compression techniques on fairness in language models
Ramesh, K., Chavan, A., Pandit, S., & Sitaram, S. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.878 · Cited by 15
Jiao, J., Sun, H., Huang, Y., Xia, M., Qiao, M., Ren, Y., Wang, Y., & Guo, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3278259 · Cited by 15
Learning few-shot imitation as cultural transmission
Bhoopchand, A., Brownfield, B., Collister, A., Dal Lago, A., Edwards, A., Everett, R., Fréchette, A., Oliveira, Y. G., Hughes, E., Mathewson, K. W., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-42875-2 · Cited by 14
Debiasing pretrained text encoders by paying attention to paying attention
Gaci, Y., Benatallah, B., Casati, F., & Benabdeslem, K. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.651 · Cited by 14
Distilling Foundation Models for Robust and Efficient Models in Digital Pathology
Filiot, A., Dop, N., Tchita, O., Riou, A., Dubois, R., Peeters, T., Valter, D., Scalbert, M., Saillard, C., Robin, G., et al. (2026) · Lecture Notes in Computer Science · doi:10.1007/978-3-032-04981-0_16 · Cited by 17
Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Zhao, A., Wu, Y., Yue, Y., Wu, T., Xu, Q., Yue, Y., Lin, M., Wang, S., Wu, Q., Zheng, Z., et al. (2025) · arXiv · Cited by 13
Metadata
- Authors
- Hinton, G., Vinyals, O., & Dean, J.
- ORCID
- Vinyals, Oriol
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1503.02531
- OpenAlex
- W1821462560
- Cited by
- 14,204
- References
- 9
- Citations by year
- 2026: 356 · 2025: 1286 · 2024: 2002 · 2023: 2430 · 2022: 1865 · 2021: 2483 · 2020: 1751 · 2019: 1033 · 2018: 555 · 2017: 270 · 2016: 115 · 2015: 51
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- knowledge distillation, neural network ensembles, model compression, ensemble averaging, mixture of experts, acoustic modeling, speech recognition, MNIST, deployment efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
