Self-Distillation Amplifies Regularization in Hilbert Space
Open accessThe DOI matches its OpenAlex record; 7 of the 7 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
Jacot, A., Gabriel, F., & Hongler, C. (2018) · arXiv · doi:10.48550/arxiv.1806.07572 · Cited by 1,562
Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, L., & Anandkumar, A. (2018) · arXiv · doi:10.48550/arxiv.1805.04770 · Cited by 1,212
Li, Z., & Hoiem, D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_37 · Cited by 659
Transferring Inductive Biases through Knowledge Distillation
Abnar, S., Dehghani, M., & Zuidema, W. (2020) · arXiv · doi:10.48550/arxiv.2006.00555 · Cited by 30
Cited by
Knowledge Distillation: A Survey
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01453-z · Cited by 3,931
Rethinking Few-Shot Image Classification: A Good Embedding is All You Need?
Tian, Y., Wang, Y., Krishnan, D., Tenenbaum, J. B., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58568-6_16 · Cited by 952
R-Drop: Regularized Dropout for Neural Networks
Liang, X., Wu, L., Li, J., Wang, Y., Meng, Q., Qin, T., Chen, W., Zhang, M., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14448 · Cited by 297
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
Kim, T., Oh, J., Kim, N. Y., Cho, S., & Yun, S.-Y. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/362 · Cited by 270
Knowledge Distillation with the Reused Teacher Classifier
Chen, D., Mei, J.-P., Zhang, H., Wang, C., Feng, Y., & Chen, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01163 · Cited by 254
“A possible explanation from self-distillation is that the feature re-representation effect in Equation (3) may help the student model become more robust and thus achieve better results [12, 35].”
ResultLeveraging recent advances in deep learning for audio-Visual emotion recognition
Schoneveld, L., Othmani, A., & Abdelkawy, H. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.03.007 · Cited by 237
“It has also been shown theoretically that self-distillation can improve a model’s performance via a regularization e ff ect (Mobahi et al. [2020]).”
BackgroundTowards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
Allen-Zhu, Z., & Li, Y. (2020) · arXiv · doi:10.48550/arxiv.2012.09816 · Cited by 178
“• (Self-distillation [36, 63] also improves test accuracy, see Figure 1): if we further train (using GD from random initialization) another network f ′ to match the output of a single model f1 merely over the same training data set Z, then f ′ can be trained efficiently and w.”
BackgroundImitation Attacks and Defenses for Black-box Machine Translation Systems
Wallace, E., Stern, M., & Song, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.446 · Cited by 75
Transferring Inductive Biases through Knowledge Distillation
Abnar, S., Dehghani, M., & Zuidema, W. (2020) · arXiv · doi:10.48550/arxiv.2006.00555 · Cited by 30
Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
Igl, M., Farquhar, G., Luketina, J., Boehmer, W., & Whiteson, S. (2020) · arXiv · doi:10.48550/arxiv.2006.05826 · Cited by 17
“This improvement has been attributed to the ease of predicting the output distribution of the teacher compared to the original ‘hard’ labels [Mobahi et al., 2020, Gotmare et al., 2019].”
ResultVerse: Virtual-gradient aware streaming lifelong learning with anytime inference
Banerjee, S., Verma, V. K., Mukherjee, A., Gupta, D., Namboodiri, V. P., & Rai, P. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610702 · Cited by 3
“Semantic memory (SEM) retains long-term knowledge and combats forgetting using self-distillation-loss [34], [53], [54], [35], [55] minimization (Eq.”
Background
Metadata
- Authors
- Mobahi, H., Farajtabar, M., & Bartlett, P. L.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2020-02-13
- Volume
- 33
- Pages
- 3351-3361
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2002.05715
- OpenAlex
- W3103361051
- Cited by
- 118
- References
- 7
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.