CoChat
Sign Up
PreprintCited by 30Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsMachine Learning and Data Classification

Transferring Inductive Biases through Knowledge Distillation

Open access
Abnar, Samira, Dehghani, Mostafa, Zuidema, Willem·2020·arXiv
DOI 10.48550/arxiv.2006.00555arXiv 2006.00555
Open this paper in CoChat — free accountOpen PDF

Abstract

Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly representative of the conditions at test time. However, defining, designing and efficiently adapting inductive biases is not necessarily straightforward. In this paper, we explore the power of knowledge distillation for transferring the effect of inductive biases from one model to another. We consider families of models with different inductive biases, LSTMs vs. Transformers and CNNs vs. MLPs, in the context of tasks and scenarios where having the right inductive biases is critical. We study the effect of inductive biases on the solutions the models converge to and investigate how and to what extent the effect of inductive biases is transferred through knowledge distillation, in terms of not only performance but also different aspects of converged solutions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.00555

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The convnet is a better teacher probably due to inductive bias inherited by transformers through distillation.

    “The fact that the convnet is a better teacher is probably due to the inductive bias inherited by the transformers through distillation, as explained in Abnar et al. [1].”

    From Training data-efficient image transformers & distillation through attention (2020) · cites this paper as [1]

    The paper supports the proposed mechanism, but not the claimed teacher–student pairing. Section 3 (p. 8) reports that a CNN is a better teacher for an MLP, while Section 2.2 (pp. 5–6) reports that LSTM teachers improve Transformers; it does not report CNNs teaching Transformers. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Knowledge distillation can transfer inductive biases from teacher to student.

    “KD can transfer inductive biases [1] in a soft way in a student model using a teacher model where they would be incorporated in a hard way.”

    From Training data-efficient image transformers & distillation through attention (2020) · cites this paper as [1]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 51

  1. Samira Abnar, Lisa Beinborn, Rochelle Choenni, and Willem Zuidema. Blackbox meets blackbox: Representational similarity & stability analysis of neural language models and brains. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , 2019. URL https://www.aclweb.org/anthology/W19-4820 .
  2. Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , CVPR'19, 2019. URL https://arxiv.org/ abs/1904.05835 . · arXiv
  3. Rohan Anil, Gabriel Pereyra, Alexandre Tachard Passos, Robert Ormandi, George Dahl, and Geoffrey Hinton. Large scale distributed neural network training through online distillation. In Proceedings of the 6th International Conference on Learning Representations , ICLR'18, 2018. URL https: //openreview.net/pdf?id=rkr1UDeC-.
  4. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020. · arXiv
  5. Cristian Buciluˇ a, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining , KDD '06, 2006. URL https://dl.acm.org/doi/10.1145/1150402.1150464 . · DOI · Paper page
  6. Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis. In Conference on Learning Theory , pp. 698-728, 2016.
  7. Mark W. Craven and Jude W. Shavlik. Extracting tree-structured representations of trained networks. In Advances in Neural Information Processing Systems 8 , NeurIPS'95, Cambridge, MA, USA, 1995. MIT Press. URL https://papers.nips.cc/paper/ 1152-extracting-tree-structured-representations-of-trained-networks. pdf .
  8. Mark William Craven. Extracting Comprehensible Models from Trained Neural Networks . PhD thesis, University of Wisconsin-Madison, 1996. URL https://www.biostat.wisc.edu/ ~craven/papers/thesis.pdf .
  9. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In Proceedings of the 7th International Conference on Learning Representations , ICLR'19, 2019. URL https://arxiv.org/abs/1807.03819 . · arXiv
  10. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv: 2002.06305 , 2020. URL https://arxiv.org/abs/2002.06305 . · arXiv
  11. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proceedings of the 7th International Conference on Learning Representations , ICLR'19, 2019. URL http://dblp.uni-trier.de/db/conf/iclr/ iclr2019.html#FrankleC19 .
  12. Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. Ensemble distillation for neural machine translation. CoRR , abs/1702.01802, 2017. URL http://arxiv.org/abs/1702.01802 . · arXiv
  13. Nicholas Frosst and Geoffrey E. Hinton. Distilling a neural network into a soft decision tree. In Proceedings of the First International Workshop on Comprehensibility and Explanation in AI and ML 2017 co-located with 16th International Conference of the Italian Association for Artificial Intelligence , 2017. URL https://arxiv.org/abs/1711.09784 . · arXiv
  14. Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural networks. In Proceedings of the 35th International Conference on Machine Learning , ICML'18, 2018. URL https://arxiv.org/abs/1805.04770 . · arXiv
  15. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, 2016. http: //www.deeplearningbook.org .
  16. Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. arXiv preprint arXiv:1302.4389 , 2013. · arXiv
  17. Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. Modeling recurrence for transformer. arXiv preprint arXiv:1904.03092 , 2019. · arXiv
  18. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015. URL https://arxiv.org/abs/1503.02531 . · arXiv
  19. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):17351780, 1997. URL https://dl.acm.org/doi/10.1162/neco.1997.9.8.1735 . · DOI · Paper page
  20. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858 , 2019. URL https://arxiv.org/abs/1909.05858 . · arXiv
  21. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014. URL https://arxiv.org/pdf/1412.6980.pdf . · arXiv
  22. Adhiguna Kuncoro, Chris Dyer, Laura Rimell, Stephen Clark, and Phil Blunsom. Scalable syntaxaware language models using knowledge distillation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 3472-3484, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1337. URL https://www. aclweb.org/anthology/P19-1337 . · DOI
  23. Adhiguna Kuncoro, Lingpeng Kong, Daniel Fried, Dani Yogatama, Laura Rimell, Chris Dyer, and Phil Blunsom. Syntactic structure distillation pretraining for bidirectional encoders. arXiv preprint arXiv:2005.13482 , 2020. · arXiv
  24. Aarre Laakso and Garrison Cottrell. Content and cluster analysis: Assessing representational similarity in neural systems. Philosophical Psychology , 13(1):47-76, 2000. doi: 10.1080/ 09515080050002726. URL https://doi.org/10.1080/09515080050002726 . · DOI
  25. Phong Le and Willem Zuidema. The inside-outside recursive neural network model for dependency parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing , EMNLP'14, 2014. URL https://www.aclweb.org/anthology/D14-1081. pdf .
  26. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of lstms to learn syntaxsensitive dependencies. Transactions of the Association for Computational Linguistics , 4(0), 2016. URL https://transacl.org/ojs/index.php/tacl/article/view/972 .
  27. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. ArXiv , abs/1904.09482, 2019. URL https://arxiv.org/abs/1904.09482 . · arXiv
  28. Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Proceedings of the 5th International Conference on Learning Representations , ICLR'17, 2017. URL https://arxiv.org/abs/1608.03983 . · arXiv
  29. Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song. Knowledge amalgamation from heterogeneous networks by common feature learning. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence , IJCAI'19, 2019. URL https://arxiv.org/abs/1906.10546 . · arXiv
  30. Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations of recurrent networks. In Advances in neural information processing systems 32 , NeurIPS'19, 2019. URL https://arxiv.org/abs/1907.08549 . · arXiv
  31. R. Thomas McCoy, Robert Frank, and Tal Linzen. Does syntax need to grow on trees? sources of hierarchical inductive bias in sequence-to-sequence networks. CoRR , abs/2001.03632, 2020. URL https://arxiv.org/abs/2001.03632 . · arXiv
  32. Hossein Mobahi, Mehrdad Farajtabar, and Peter L Bartlett. Self-distillation amplifies regularization in hilbert space. arXiv preprint arXiv:2002.05715 , 2020. · arXiv
  33. Norman Mu and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision. arXiv preprint arXiv:1906.02337 , 2019. · arXiv
  34. Rafael Rodrigo Mueller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems 32 , NeurIPS'19, 2019. URL https://arxiv.org/abs/1906.02629 . · arXiv
  35. Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? arXiv preprint arXiv:2008.11687 , 2020. · arXiv
  36. Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition , CVPR'19, 2019. URL https://arxiv.org/abs/1904.05068 . · arXiv
  37. Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. volume 97 of Proceedings of Machine Learning Research , pp. 5142-5151, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http://proceedings.mlr.press/v97/phuong19a.html .
  38. Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In Proceedings of the 6th International Conference on Learning Representations , ICLR'18, 2018. URL https://openreview.net/forum?id=ry\_WPG-A-.
  39. H. S. Seung, H. Sompolinsky, and N. Tishby. Learning curves in large neural networks. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory , COLT '91, 1991. URL https://dl.acm.org/doi/10.5555/114836.114847 . · DOI
  40. Richard Socher, Christopher D. Manning, and Andrew Y. Ng. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In In Proceedings of the NeurIPS'10 Deep Learning and Unsupervised Feature Learning Workshop , 2010. URL https://ai.stanford.edu/~ang/papers/ nipsdlufl10-LearningContinuousPhraseRepresentations.pdf .
  41. Suraj Srinivas and R. Venkatesh Babu. Data-free parameter pruning for deep neural networks. In Proceedings of the 26th British Machine Vision Conference , BMVC'15, 2015. URL https://www.semanticscholar.org/paper/ Data-free-Parameter-Pruning-for-Deep-Neural-Srinivas-Babu/ b0bd441a0cc04cdd0d0e469fe4c5184ee148a97d .
  42. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. In EMNLP/IJCNLP , 2019.
  43. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning , ICML'13, 2013. URL https://dl.acm.org/doi/10.5555/3042817.3043064 . · DOI
  44. Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. Multilingual neural machine translation with knowledge distillation. arXiv preprint arXiv:1902.10461 , 2019. URL https: //arxiv.org/abs/1902.10461 . · arXiv
  45. Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and improving knowledge distillation. arXiv preprint arXiv:2002.03532 , 2020. · arXiv
  46. Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling taskspecific knowledge from bert into simple neural networks. ArXiv , abs/1903.12136, 2019. URL https://arxiv.org/abs/1903.12136 . · arXiv
  47. Ke Tran, Arianna Bisazza, and Christof Monz. The importance of being recurrent for modeling hierarchical structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , EMNLP'18, 2018. URL https://www.aclweb.org/anthology/ D18-1503 .
  48. Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. ArXiv , abs/1907.09682, 2019. URL https://arxiv.org/abs/1907.09682 . · arXiv
  49. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems 30 , NeurIPS'17, pp. 5998-6008, 2017. URL https://arxiv.org/abs/ 1706.03762 .
  50. D. H. Wolpert and W. G. Macready. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation , 1(1):67-82, April 1997. ISSN 1089-778X. doi: 10.1109/4235.585893. URL https://doi.org/10.1109/4235.585893 . · DOI · Paper page
  51. Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisit knowledge distillation: a teacher-free framework. arXiv preprint arXiv:1909.11723 , 2019. · arXiv

Cited by

Sorted by citations

Authors
Abnar, S., Dehghani, M., & Zuidema, W.
Affiliations
  • University of Amsterdam (NL)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.00555
arXiv
2006.00555
OpenAlex
W3031955466
Cited by
30
References
7
Citations by year
2025: 3 · 2024: 4 · 2023: 9 · 2022: 5 · 2021: 4 · 2020: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Nederlandse Organisatie voor Wetenschappelijk Onderzoek (024.001.006)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
inductive bias, knowledge distillation, LSTM, transformer models, convolutional neural networks, MLP, model convergence, transfer learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review