CoChat
Sign Up
PreprintCited by 44Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningAdvanced Image and Video Retrieval Techniques

The DOI matches its DataCite record; its reference list is not available.

Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation

Open access
Hao, Zhiwei, Guo, Jianyuan, Jia, Ding, Han, Kai, Tang, Yehui, Zhang, Chao, Hu, Han, Wang, Yunhe·2021·arXiv
DOI 10.48550/arxiv.2107.01378arXiv 2107.01378
Open this paper in CoChat — free accountOpen PDF

Abstract

In the past few years, transformers have achieved promising performances on various computer vision tasks. Unfortunately, the immense inference overhead of most existing vision transformers withholds their from being deployed on edge devices such as cell phones and smart watches. Knowledge distillation is a widely used paradigm for compressing cumbersome architectures via transferring information to a compact student. However, most of them are designed for convolutional neural networks (CNNs), which do not fully investigate the character of vision transformer (ViT). In this paper, we utilize the patch-level information and propose a fine-grained manifold distillation method. Specifically, we train a tiny student model to match a pre-trained teacher model in the patch-level manifold space. Then, we decouple the manifold matching loss into three terms with careful design to further reduce the computational costs for the patch relationship. Equipped with the proposed method, a DeiT-Tiny model containing 5M parameters achieves 76.5% top-1 accuracy on ImageNet-1k, which is +2.0% higher than previous distillation approaches. Transfer learning results on other classification benchmarks and downstream vision tasks also demonstrate the superiority of our method over the state-of-the-art algorithms.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2107.01378

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Vision Transformers suffer from higher memory footprints, computational overheads, and power consumption than convolutional neural networks, hindering their deployment and real-time inference on resource-constrained edge devices.

    “Nevertheless, as compared to convolutional neural networks (CNNs), ViTs suffer from higher memory footprints, computational overheads, and power consumption, hindering their deployment and real-time inference on resourceconstrained edge devices [25, 17, 15, 37].”

    From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [15]

    The Introduction (pp. 1–2) says vision transformers often require more parameters and substantial computing resources and storage, and that these requirements impede deployment on memory-bounded edge devices. It compares an 86M-parameter ViT-B with a 5M-parameter CNN, but does not establish that ViTs generally consume more power than CNNs or specifically hinder real-time inference. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 44

Research works printed but not linked: 31

  1. [1][1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics , 2019. -
  2. [2][2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems , 2020. -
  3. [3][3] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, et al. Attention is all you need. In Advances in Neural Information Processing Systems , 2017. -
  4. [4][4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021. -
  5. [5][5] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, et al. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision , 2021. -
  6. [6][6] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, et al. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In International Conference on Computer Vision , 2021. -
  7. [7][7] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, et al. Twins: Revisiting the design of spatial attention in vision transformers. In Advances in Neural Information Processing Systems , 2021. -
  8. [8][8] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. In Advances in Neural Information Processing Systems , 2021. -
  9. [9][9] Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning , 2019. -
  10. [10][10] Mingjian Zhu, Kai Han, Yehui Tang, and Yunhe Wang. Visual transformer pruning. arXiv preprint arXiv:2104.08500 , 2021. - · arXiv
  11. [11][11] Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, et al. Patch slimming for efficient vision transformers. arXiv preprint arXiv:2106.02852 , 2021. - · arXiv
  12. [13][13] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015. - · arXiv
  13. [14][14] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning , 2021. -
  14. [15][15] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. In Conference on Empirical Methods in Natural Language Processing , 2019. -
  15. [16][16] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, et al. Tinybert: Distilling BERT for natural language understanding. In Conference on Empirical Methods in Natural Language Processing , 2020. -
  16. [17][17] Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. Learning student networks via feature embedding. IEEE Trans. Neural Networks Learn. Syst. , 32(1):25-35, 2021. -
  17. [18][18] Baoyun Peng, Xiao Jin, Dongsheng Li, Shunfeng Zhou, Yichao Wu, Jiaheng Liu, et al. Correlation congruence for knowledge distillation. In International Conference on Computer Vision , 2019. -
  18. [19][19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, et al. Attention is all you need. In Advances in Neural Information Processing Systems , 2017. -
  19. [20][20] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision , 2020. -
  20. [21][21] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, et al. Generative pretraining from pixels. In International Conference on Machine Learning , 2020. -
  21. [22][22] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Qibin Hou, et al. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886 , 2021. - · arXiv
  22. [23][23] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In International Conference on Computer Vision , 2021. -
  23. [24][24] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, et al. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In International Conference on Computer Vision , 2021. -
  24. [25][25] Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. Adversarial learning of portable student networks. In AAAI Conference on Artificial Intelligence , 2018. -
  25. [26][26] Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, et al. Positiveunlabeled compression on the cloud. In Advances in Neural Information Processing Systems , 2019. -
  26. [27][27] Guorui Zhou, Ying Fan, Runpeng Cui, Weijie Bian, Xiaoqiang Zhu, and Kun Gai. Rocket launching: A universal and efficient framework for training well-performing light net. In AAAI Conference on Artificial Intelligence , 2018. -
  27. [28][28] Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science , 290(5500):2323-2326, 2000. -
  28. [29][29] Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science , 290(5500):2319-2323, 2000. -
  29. [30][30] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference on Learning Representations , 2015. -
  30. [31][31] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision , 2016. -
  31. [35][35] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, et al. Microsoft COCO: common objects in context. In European Conference on Computer Vision , Lecture Notes in Computer Science, 2014. -

Metadata

Authors
Hao, Z., Guo, J., Jia, D., Han, K., Tang, Y., Zhang, C., Hu, H., & Wang, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
9164-9175
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2107.01378
arXiv
2107.01378
OpenAlex
W7133185110
Cited by
44
References
0
Citations by year
2026: 2 · 2025: 15 · 2024: 19 · 2023: 8
FWCI
3.81
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
Keywords
vision transformer, knowledge distillation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review