CoChat
Sign Up
PreprintCited by 951Multimodal Machine Learning ApplicationsAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

Open access
Kim, Wonjae, Son, Bokyung, Kim, Ildoo·2021·arXiv
DOI 10.48550/arxiv.2102.03334arXiv 2102.03334
Open this paper in CoChat — free accountOpen PDF

Abstract

Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2102.03334

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing multimodal models typically assume that all modalities are available during both training and inference.

    “Existing multimodal models typically assume that all modalities are available during both training and inference [Kim et al. , 2021].”

    From Enhancing Multimodal Model Robustness Under Missing Modalities via Memory-Driven Prompt Learning (2025) · cites this paper as Kim et al., 2021

    The Introduction (p. 1) says vision-and-language pre-training models use images with aligned descriptions and are fine-tuned on tasks whose inputs involve both modalities. This supports the claim for the vision-and-language models discussed, but the paper does not establish that multimodal models generally assume every modality is available during both training and inference. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • In cross-modal self-supervised learning, tasks such as image inpainting or text mask prediction are typically designed.

    “In cross-modal self-supervised learning, tasks such as image inpainting or text mask prediction are typically designed.[183]”

    From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [183]

    The Introduction (p. 1) and Pre-training Objectives (p. 5) describe masked language modeling as common in vision-and-language pre-training, including prediction of masked text tokens. They identify image–text matching—not image inpainting—as the other common objective. The Conclusion (p. 9) calls for future work on visual masking objectives, so the paper does not support presenting image inpainting as a typical task. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Existing solutions in multimodal Transformer adopt a fixed fusion strategy.

    “Existing solutions in multimodal Transformer adopt a fixed fusion strategy [18,21,24,26,33].”

    From Are Multimodal Transformers Robust to Missing Modality? (2022) · cites this paper as [18]

    Section 2.2 (p. 3) describes established multimodal Transformer models as using single-stream or dual-stream interaction schemas, and says ViLT chooses the single-stream approach. This supports the claim that models use defined fusion architectures, but the paper does not establish that their fusion strategies are fixed rather than adaptive. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • The training load of conventional multimodal backbones grows as the number of modalities increases, whereas Transformer models process all modalities simultaneously using a single model, greatly reducing training load.

    “The training load of conventional multimodal backbone grows as the number of modalities increases since the backbone usually consists of modalityspecific sub-models that need to be trained independently for each modality [27, 41]. Instead, Transformer models process all modalities simultaneously using a single model [18,24], which greatly reduces the training load.”

    From Are Multimodal Transformers Robust to Missing Modality? (2022) · cites this paper as [18]

    The Introduction (p. 2) says ViLT handles image and text in a unified model and reduces model size and runtime by removing a separate deep visual embedder. Section 2.2 (p. 3) describes its single-stream transformer, but also notes that some vision-and-language transformers use dual streams. The paper does not establish that transformers generally process all modalities in one model or that doing so greatly reduces training load as the number of modalities grows. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • ViLT represents the common design of multimodal transformer.

    “We use ViLT as the backbone since it represents the common design of multimodal transformer.”

    From Are Multimodal Transformers Robust to Missing Modality? (2022) · cites this paper as [18]

    Section 2.2 (p. 3) says ViLT follows the established single-stream transformer approach, supporting its use as an example of a multimodal transformer. But Section 2.1 (p. 3) calls ViLT the first model of its architectural type, and the Introduction (p. 2) emphasizes how its visual embedding differs from previous models. The source therefore does not support describing ViLT's overall design as common. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 6077-6086, 2018.
  2. Asano, Y., Rupprecht, C., and Vedaldi, A. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations , 2019.
  3. Berman, M., Jégou, H., Vedaldi, A., Kokkinos, I., and Douze, M. Multigrain: a unified image embedding for classes and instances. arXiv preprint arXiv:1902.05509 , 2019. · arXiv
  4. Bugliarello, E., Cotterell, R., Okazaki, N., and Elliott, D. Multimodal pretraining unmasked: Unifying the vision and language berts. arXiv preprint arXiv:2011.15124 , 2020. · arXiv
  5. Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV) , pp. 132-149, 2018.
  6. Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882 , 2020. · arXiv
  7. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pp. 1597-1607. PMLR, 2020a.
  8. Chen, X., Fan, H., Girshick, R., and He, K. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020b. · arXiv
  9. Chen, Y.-C., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. Uniter: Learning universal imagetext representations. arXiv preprint arXiv:1909.11740 , 2019. · arXiv
  10. Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pp. 702-703, 2020.
  11. Cui, Y., Che, W., Liu, T., Qin, B., Yang, Z., Wang, S., and Hu, G. Pre-training with whole word masking for chinese bert. arXiv preprint arXiv:1906.08101 , 2019. · arXiv
  12. Faghri, F., Fleet, D. J., Kiros, J. R., and Fidler, S. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612 , 2017. · arXiv
  13. Gan, Z., Chen, Y.-C., Li, L., Zhu, C., Cheng, Y., and Liu, J. Large-scale adversarial training for vision-and-language representation learning. arXiv preprint arXiv:2006.06195 , 2020. · arXiv
  14. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 6904-6913, 2017.
  15. He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask rcnn. In Proceedings of the IEEE international conference on computer vision , pp. 2961-2969, 2017.
  16. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 97299738, 2020.
  17. Hoffer, E., Ben-Nun, T., Hubara, I., Giladi, N., Hoefler, T., and Soudry, D. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 8129-8138, 2020.
  18. Huang, Z., Zeng, Z., Liu, B., Fu, D., and Fu, J. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849 , 2020. · arXiv
  19. Jiang, H., Misra, I., Rohrbach, M., Learned-Miller, E., and Chen, X. In defense of grid features for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1026710276, 2020.
  20. Jiang, Y., Natarajan, V., Chen, X., Rohrbach, M., Batra, D., and Parikh, D. Pythia v0. 1: the winning entry to the vqa challenge 2018. arXiv preprint arXiv:1807.09956 , 2018. · arXiv
  21. Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. The open images dataset v4. International Journal of Computer Vision , pp. 1-26, 2020.
  22. Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., and Chang, K.-W. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 , 2019. · arXiv
  23. Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al. Oscar: Objectsemantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision , pp. 121137. Springer, 2020b.
  24. Common objects in context. In European conference on computer vision , pp. 740-755. Springer, 2014.
  25. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.
  26. Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pretraining task-agnostic visiolinguistic representations for visionand-language tasks. In Advances in Neural Information Processing Systems , pp. 13-23, 2019.
  27. Lu, J., Goswami, V., Rohrbach, M., Parikh, D., and Lee, S. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1043710446, 2020.
  28. Nguyen, D.-K., Goswami, V., and Chen, X. Revisiting modulated convolutions for visual counting and beyond. arXiv preprint arXiv:2004.11883 , 2020. · arXiv
  29. Ordonez, V., Kulkarni, G., and Berg, T. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems , 24:1143-1151, 2011.
  30. Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision , pp. 26412649, 2015.
  31. Qi, D., Su, L., Song, J., Cui, E., Bharti, T., and Sacheti, A. Imagebert: Cross-modal pre-training with largescale weak-supervised image-text data. arXiv preprint arXiv:2001.07966 , 2020. · arXiv
  32. Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 25562565, 2018.
  33. Shorten, C. and Khoshgoftaar, T. M. A survey on image data augmentation for deep learning. Journal of Big Data , 6(1):1-48, 2019.
  34. Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., and Dai, J. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 , 2019. · arXiv
  35. Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., and Artzi, Y. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811.00491 , 2018. · arXiv
  36. Tan, H. and Bansal, M. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490 , 2019. · arXiv
  37. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877 , 2020. · arXiv
  38. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems , 30:5998-6008, 2017.
  39. Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 1492-1500, 2017.
  40. Xie, Y., Wang, X., Wang, R., and Zha, H. A fast proximal point method for computing exact wasserstein distance. In Uncertainty in Artificial Intelligence , pp. 433-453. PMLR, 2020.
  41. Yu, F., Tang, J., Yin, W., Sun, Y., Tian, H., Wu, H., and Wang, H. Ernie-vil: Knowledge enhanced visionlanguage representations through scene graph. arXiv preprint arXiv:2006.16934 , 2020. · arXiv
  42. Yu, Z., Yu, J., Cui, Y ., Tao, D., and Tian, Q. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 6281-6290, 2019.
  43. Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 6023-6032, 2019.
  44. Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017. · arXiv
  45. Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., and Gao, J. Vinvl: Making visual representations matter in vision-language models. arXiv preprint arXiv:2101.00529 , 2021. · arXiv
  46. Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J. J., and Gao, J. Unified vision-language pre-training for image captioning and vqa. In AAAI , pp. 13041-13049, 2020.

Cited by

Sorted by citations

Authors
Kim, W., Son, B., & Kim, I.
Affiliations
NAVER Cloud (South Korea) (KR)
ORCID
Kim, Wonjae
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2102.03334
arXiv
2102.03334
OpenAlex
W3126792443
Cited by
951
References
13
Citations by year
2026: 5 · 2025: 173 · 2024: 304 · 2023: 332 · 2022: 118 · 2021: 19
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
image embeddings, multimodal interaction, computational efficiency, pre-trained language models, vision-language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review