ConViT: improving vision transformers with soft convolutional inductive biases
Open accessAbstract
Convolutional architectures have proven to be extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling. Vision transformers rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification. However, they require costly pre-training on large external datasets or distillation from pre-trained convolutional networks. In this paper, we ask the following question: is it possible to combine the strengths of these two architectures while avoiding their respective limitations? To this end, we introduce gated positional self-attention (GPSA), a form of positional self-attention which can be equipped with a ‘soft’ convolutional inductive bias. We initialize the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information. The resulting convolutional-like ViT architecture, ConViT , outperforms the DeiT (Touvron et al 2020 arXiv: 2012.12877 ) on ImageNet, while offering a much improved sample efficiency. We further investigate the role of locality in learning by first quantifying how it is encouraged in vanilla self-attention layers, then analyzing how it has escaped in GPSA layers. We conclude by presenting various ablations to better understand the success of the ConViT. Our code and models are released publicly at https://github.com/facebookresearch/convit .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RPE is often jointly used with absolute positional encodings, or the absolute position information is leaked in other ways.
“Additionally, although RPE becomes popular recently, it is often jointly used with absolute positional encodings (e.g., in ConViT (d'Ascoli et al., 2021)), or the absolute position information is leaked in other ways (e.g., convolution paddings in CoAtNet (Dai et al., 2021)).”
From Conditional Positional Encodings for Vision Transformers (2021) · cites this paper as d'Ascoli et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · cited 1× · see citing works
“A breadth of approaches have been taken to imbue CNN architectures with nonlocality [34, 35, 36, 28].”
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955 · see citing works
Natural Image Statistics and Neural Representation
Simoncelli, E. P., & Olshausen, B. A. (2001) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.24.1.1193 · Cited by 2,460 · see citing works
Flagged first, then by citations
Research works printed but not linked: 36
- Abnar S, Dehghani M and Zuidema W 2020 Transferring inductive biases through knowledge distillation (arXiv:2006.00555) · arXiv
- Anandkumar A, Deng Y, Ge R and Mobahi H 2016 Homotopy analysis for tensor PCA (arXiv:1610.09322) · arXiv
- Bahdanau D, Cho K and Bengio Y 2014 Neural machine translation by jointly learning to align and translate (arXiv:1409.0473) · arXiv
- Bello I, Zoph B, Vaswani A, Shlens J and Le Q V 2019 Attention augmented convolutional networks Proc. IEEE Int. Conf. Computer Vision pp 3286-95
- Chen Y, Kalantidis Y, Li J, Yan S and Feng J 2018 A2-nets: double attention networks (arXiv:1810.11579) · arXiv
- Chen Y-C, Li L, Yu L, El Kholy A, Ahmed F, Gan Z, Cheng Y and Liu J U 2020 UNITER: UNiversal image-TExt representation learning European Conf. Computer Vision (Springer) pp 104-20
- Cordonnier J-B, Loukas A and Jaggi M 2019 On the relationship between self-attention and convolutional layers (arXiv:1911.03584) · arXiv
- 'Ascoli S, Sagun L, Biroli G and Bruna J 2019 Finding the needle in the haystack with convolutions: on the benefits of architectural bias Advances in Neural Information Processing Systems pp 9334-45
- Devlin J, Chang M-W, Lee K and Toutanova K 2018 BERT: pre-training of deep bidirectional transformers for language understanding (arXiv:1810.04805) · arXiv
- Dong Y, Cordonnier J-B and Loukas A 2021 Attention is not all you need: pure attention loses rank doubly exponentially with depth (arXiv:2103.03404) · arXiv
- Elsayed G, Ramachandran P, Shlens J and Kornblith S 2020 Revisiting spatial invariance with low-rank local connectivity Int. Conf. Machine Learning (PMLR) pp 2868-79
- Goodfellow I, Bengio Y and Courville A 2016 Deep Learning (Cambridge, MA: MIT Press)
- Hinton G, Vinyals O and Dean J 2015 Distilling the knowledge in a neural network (arXiv:1503.02531) · arXiv
- Hu H, Gu J, Zhang Z, Dai J and Wei Y 2018a Relation networks for object detection Proc. IEEE Conf. Computer Vision and Pattern Recognition pp 3588-97
- Hu J, Shen L, Albanie S, Sun G and Vedaldi A 2018b Gather-excite: exploiting feature context in convolutional neural networks Advances in Neural Information Processing Systems vol 31 ed S Bengio, H Wallach, H Larochelle, K Grauman, N Cesa-Bianchi and R Garnett (Curran Associates) pp 9401-11
- Hu J, Shen L and Sun G 2018c Squeeze-and-Excitation networks 2018 IEEE/CVF Conf. Computer Vision and Pattern Recognition pp 7132-41
- Lecun Y, Bottou L, Bengio Y and Haffner P 1998 Gradient-based learning applied to document recognition Proc. IEEE 86 2278-324 · DOI · Paper page
- Locatello F, Weissenborn D, Unterthiner T, Mahendran A, Heigold G, Uszkoreit J, Dosovitskiy A and Kipf T 2020 Object-centric learning with slot attention (arXiv:2006.15055) · arXiv
- Neyshabur B 2020 Towards learning convolutions from scratch Advances in Neural Information Processing Systems vol 33
- Radosavovic I, Kosaraju R P, Girshick R, He K and Dollár P 2020 Designing network design spaces Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition pp 10428-36
- Ramachandran P, Parmar N, Vaswani A, Bello I, Levskaya A and Shlens J 2019 Stand-alone self-attention in vision models (arXiv:1906.05909) · arXiv
- Ruderman D L and Bialek W 1994 Statistics of natural images: scaling in the woods Phys. Rev. Lett. 73 814 · DOI
- Scherer D, Müller A and Behnke S 2010 Evaluation of pooling operations in convolutional architectures for object recognition Artificial Neural Networks-ICANN 2010 Lecture Notes in Computer Science ed K Diamantaras, W Duch and L S Iliadis (Berlin: Springer) pp 92-101
- Schmidhuber J 2015 Deep learning in neural networks: an overview Neural Netw. 61 85-117 · DOI · Paper page
- Srinivas A, Lin T-Y, Parmar N, Shlens J, Abbeel P and Vaswani A 2021 Bottleneck transformers for visual recognition (arXiv:2101.11605) · arXiv
- Sukhbaatar S, Grave E, Bojanowski P and Joulin A 2019 Adaptive attention span in transformers (arXiv:1905.07799) · arXiv
- Sun C, Myers A, Vondrick C, Murphy K and Schmid C V 2019 A joint model for video and language representation learning Proc. IEEE Int. Conf. Computer Vision pp 7464-73
- Sundermeyer M, Schlüter R and Ney H 2012 LSTM neural networks for language modeling 13th Annual Conf. Int. Speech Communication Association
- Tolstikhin I et al 2021 MLP-mixer: an all-MLP architecture for vision (arXiv:2105.01601) · arXiv
- Touvron H, Cord M, Douze M, Massa F, Sablayrolles A and Jégou H 2020 Training data-efficient image transformers & distillation through attention (arXiv:2012.12877) · arXiv
- Touvron H et al 2021a ResMLP: feedforward networks for image classification with data-efficient training (arXiv:2105.03404) · arXiv
- Touvron H, Cord M, Sablayrolles A, Synnaeve G and Jégou H 2021b Going deeper with image transformers Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser L and Polosukhin I 2017 Attention is all you need Advances in Neural Information Processing Systems pp 5998-6008
- Wu B, Xu C, Dai X, Wan A, Zhang P, Tomizuka M, Keutzer K and Vajda P 2020 Visual transformers: token-based image representation and processing for computer vision (arXiv:2006.03677) · arXiv
- Yuan L, Chen Y, Wang T, Yu W, Shi Y, Tay F E, Feng J and Yan S 2021 Tokens-to-token ViT: training vision transformers from scratch on imagenet (arXiv:2101.11986) · arXiv
- Zhai X, Oliver A, Kolesnikov A and Beyer L 2019 S4L: self-supervised semi-supervised learning Proc. IEEE/CVF Int. Conf. Computer Vision pp 1476-85
- Zhao S, Zhou L, Wang W, Cai D, Lam T L and Xu Y 2020 Splitnet: divide and co-training (arXiv:2011.14660) https://doi.org/10.1088/1742-5468/ac9830 · arXiv
Cited by
Sorted by citations
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01167 · Cited by 8,494 · see citing works
Mobile-Former: Bridging MobileNet and Transformer
Chen, Y., Dai, X., Chen, D., Liu, M., Dong, X., Yuan, L., & Liu, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00520 · Cited by 731 · see citing works
Conditional Positional Encodings for Vision Transformers
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489 · see citing works
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
Bannur, S., Hyland, S., Liu, Q., Pérez-García, F., Ilse, M., Castro, D. C., Boecking, B., Sharma, H., Bouzid, K., Thieme, A., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01442 · Cited by 227 · see citing works
How Do Vision Transformers Work?
Park, N., & Kim, S. (2022) · arXiv · doi:10.48550/arxiv.2202.06709 · Cited by 356 · see citing works
“This is contrary to the popular belief that MSAs closer to the end of a model improve the performance (Srinivas et al., 2021; d’Ascoli et al., 2021; Graham et al., 2021; Dai et al., 2021).”
BackgroundResultHRFormer: High-Resolution Transformer for Dense Prediction
Yuan, Y., Fu, R., Huang, L., Lin, W., Zhang, C., Chen, X., & Wang, J. (2021) · arXiv · doi:10.48550/arxiv.2110.09408 · Cited by 178 · see citing works
Do Vision Transformers See Like Convolutional Neural Networks?
Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., & Dosovitskiy, A. (2021) · arXiv · doi:10.48550/arxiv.2108.08810 · Cited by 663 · see citing works
A Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78 · see citing works
Attention Weighted Local Descriptors
Wang, C., Xu, R., Lu, K., Xu, S., Meng, W., Zhang, Y., Fan, B., & Zhang, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3266728 · Cited by 36 · see citing works
Vision Transformers: From Semantic Segmentation to Dense Prediction
Zhang, L., Lu, J., Zheng, S., Zhao, X., Zhu, X., Fu, Y., Xiang, T., Feng, J., & Torr, P. H. S. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02173-w · Cited by 23 · see citing works
Triple Robustness Augmentation Local Features for multi-source image registration
Wang, C., Xu, L., Xu, R., Xu, S., Meng, W., Wang, R., & Zhang, X. (2023) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2023.03.023 · Cited by 22 · see citing works
What Do Self-Supervised Vision Transformers Learn?
Park, N., Kim, W., Heo, B., Kim, T., & Yun, S. (2023) · arXiv · doi:10.48550/arxiv.2305.00729 · Cited by 29 · see citing works
“…features—e.g., local patterns, high-frequency signals, and texture information—and later layers capture global patterns, low-frequency signals, and shape information (Dosovitskiy et al., 2021; Raghu et al., 2021; d’Ascoli et al., 2021; Graham et al., 2021; Dai et al., 2021; Park & Kim, 2022b).”
Background
- Authors
- d’Ascoli, S., Touvron, H., Leavitt, M. L., Morcos, A. S., Biroli, G., & Sagun, L.
- Affiliations
- École Normale Supérieure - PSL (FR)
- Meta (United States) (US)
- Corresponding author
- d’Ascoli, Stéphane
- Leavitt, Matthew L
- Venue
- Journal of Statistical Mechanics: Theory and Experiment
- Publisher
- IOP Publishing
- Published
- 2022-11-01
- Volume
- 2022
- Issue
- 11
- Pages
- 114005
- ISSN
- 1742-5468 (ISSN-L)
- OpenAlex
- W3137278571
- Cited by
- 782
- References
- 9
- Citations by year
- 2026: 65 · 2025: 137 · 2024: 189 · 2023: 210 · 2022: 135 · 2021: 46
- FWCI
- 54.88
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Statistical Mechanics Theory and Experiment (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Agence Nationale de la Recherche (ANR‐19, ANR-19-P3IA-0001)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- vision transformer, locality, sample efficiency, attention heads, image classification, CNN-Transformer hybrid, pre-training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
