CoChat
Sign Up
Cited by 155Multimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot LearningAdvanced Neural Network Applications

The DOI matches its Crossref record; 291 of the 291 references it cites were found in a registry or an index.

Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends

Open access
Gan, Zhe, Li, Linjie, Li, Chunyuan, Wang, Lijuan, Liu, Zicheng, Gao, Jianfeng·2022·Foundations and Trends® in Computer Graphics and Vision
DOI 10.1561/0600000105
Open this paper in CoChat — free account

Abstract

This monograph surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years. We group these approaches into three categories: (i) VLP for image-text tasks, such as image captioning, image-text retrieval, visual question answering, and visual grounding; (ii) VLP for core computer vision tasks, such as (open-set) image classification, object detection, and segmentation; and (iii) VLP for video-text tasks, such as video captioning, video-text retrieval, and video question answering. For each category, we present a comprehensive review of state-of-the-art methods, and discuss the progress that has been made and challenges still being faced, using specific systems and models as case studies. In addition, for each category, we discuss advanced topics being actively explored in the research community, such as big foundation models, unified modeling, in-context few-shot learning, knowledge, robustness, and computer vision in the wild, to name a few.

Overview

Integrity signals

Registry

Crossref · 10.1561/0600000105

Cites

291 of 291 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 242

  1. Abu-El-Haija, 2016, Youtube-8m: A largescale video classification benchmark
  2. Aghajanyan, 2022, Cm3: A causal masked multimodal model of the internet
  3. Agrawal, 2022, Rethinking evaluation practices in visual question answering: A case study on out-of-distribution generalization
  4. Akbari, 2021, Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text
  5. Alayrac, 2020, Self-supervised multimodal versatile networks
  6. Baevski, 2020, Wav2vec 2.0: A framework for self-supervised learning of speech representations
  7. Bahdanau, 2015, Neural machine translation by jointly learning to align and translate
  8. Bain, 2021, Frozen in time: A joint video and image encoder for end-to-end retrieval, 10.1109/ICCV48922.2021.00175 · DOI · Paper page
  9. Banerjee, 2005, METEOR: An automatic metric for MT evaluation with improved correlation with human judgments
  10. Bao, 2022, BEiT: BERT pre-training of image transformers
  11. Bao, 2022, VL-BEiT: Generative vision-language pretraining
  12. Bianchi, 2021, Contrastive language-image pre-training for the italian language
  13. Birhane, 2021, Multimodal datasets: Misogyny, pornography, and malignant stereotypes
  14. Bitton, 2022, WinoGAViL: Gamified association benchmark to challenge vision-and-language models
  15. Brown, 2020, Language models are few-shot learners
  16. Bugliarello, 2021, Multimodal pretraining unmasked: Unifying the vision and language BERTs
  17. Cadene, 2019, RUBi: Reducing unimodal biases for visual question answering
  18. Carlsson, 2022, Crosslingual and multilingual CLIP
  19. Chen, 2011, Collecting highly parallel data for paraphrase evaluation
  20. Chen, 2022, VLP: A survey on vision-language pre-training
  21. Chen, 2020, Graph optimal transport for cross-domain alignment
  22. Chen, 2019, Improving sequence-to-sequence learning via optimal transport
  23. Chen, 2020, Generative pretraining from pixels
  24. Chen, 2020, The lottery ticket hypothesis for pre-trained bert networks
  25. Chen, 2022, Pix2seq: A language modeling framework for object detection
  26. Chen, 2022, Pali: A jointly-scaled multilingual language-image model
  27. Chen, 2015, Microsoft COCO captions: Data collection and evaluation server
  28. Chen, 2018, Real-time referring expression comprehension by single-stage grounding network
  29. Cho, 2021, Unifying vision-and- language tasks via text generation
  30. Chowdhery, 2022, Palm: Scaling language modeling with pathways
  31. Clark, 2020, Electra: Pre-training text encoders as discriminators rather than generators
  32. Dai, 2022, One model, multiple modalities: A sparsely activated approach for text, sound, image, video and code
  33. Damen, 2018, Scaling egocentric vision: The epic-kitchens dataset
  34. Desai, 2021, Redcaps: Web-curated image-text data created by the people, for the people
  35. Devlin, 2019, Bert: Pre-training of deep bidirectional transformers for language understanding
  36. Dhariwal, 2021, Diffusion models beat gans on image synthesis
  37. Diao, 2022, Prefix language models are unified modal learners
  38. Ding, 2021, Cogview: Mastering text-to-image generation via transformers
  39. Ding, 2022, Open-vocabulary panoptic segmentation with MaskCLIP
  40. Dosovitskiy, 2021, An image is worth 16 × 16 words: Transformers for image recognition at scale
  41. Esser, 2021, Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis
  42. Faghri, 2017, Vse++: Improving visual-semantic embeddings with hard negatives
  43. Fang, 2022, Data determines distributional robustness in contrastive language image pre-training (clip)
  44. Feng, 2022, Promptdet: Expand your detector vocabulary with uncurated images
  45. Frankle, 2019, The lottery ticket hypothesis: Finding sparse, trainable neural networks
  46. Frome, 2013, Devise: A deep visual-semantic embedding model
  47. Fu, 2021, VIOLET: End-to-end video-language transformers with masked visual-token modeling
  48. Fu, 2022, An empirical study of end-to-end video-language transformers with masked visual modeling
  49. Gan, 2020, Largescale adversarial training for vision-and-language representation learning
  50. Gan, 2019, Multistep reasoning via recurrent dual attention for visual dialog
  51. Gao, 2015, Are you talking to a machine? dataset and methods for multilingual image question
  52. Gao, 2022, Neural approaches to conversational information retrieval
  53. Gao, 2021, CLIP-adapter: Better vision-language models with feature adapters
  54. Geng, 2022, Multimodal masked autoencoders learn transferable representations
  55. Ghiasi, 2022, Open-vocabulary image segmentation
  56. Goel, 2022, Cyclip: Cyclic contrastive language-image pretraining
  57. Grigoryev, 2022, When, why, and which pretrained GANs are useful?
  58. Gu, 2022, Wukong: 100 million largescale chinese cross-modal pre-training dataset and a foundation framework
  59. Gu, 2022, Open-vocabulary object detection via vision and language knowledge distillation
  60. Gupta, 2022, GRIT: General robust image task benchmark
  61. Gutmann, 2010, Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
  62. Guu, 2020, Retrieval augmented language model pre-training
  63. Hao, 2020, Towards learning a generic agent for vision-and-language navigation via pretraining
  64. Hao, 2022, Language models are general-purpose interfaces
  65. He, 2021, DeBERTa: Decoding-enhanced bert with disentangled attention
  66. He, 2022, Parameterefficient fine-tuning for vision transformers
  67. Herdade, 2019, Image captioning: Transforming objects into words
  68. Ho, 2022, Imagen video: High definition video generation with diffusion models
  69. Ho, 2020, Denoising diffusion probabilistic models
  70. Hoffmann, 2022, Training compute-optimal large language models
  71. Houlsby, 2019, Parameterefficient transfer learning for NLP
  72. Hu, 2021, Lora: Low-rank adaptation of large language models
  73. Hu, 2022, Scaling up vision-language pre-training for image captioning
  74. Huang, 2020, Pixel-BERT: Aligning image pixels with text by deep multi-modal transformers
  75. Hudson, 2018, Compositional attention networks for machine reasoning
  76. Hudson, 2019, Learning by abstraction: The neural state machine
  77. Huo, 2021, Wenlan: Bridging vision and language by large-scale multi-modal pre-training
  78. Ilharco, 2021, Openclip
  79. Jain, 2021, Mural: Multimodal, multitask retrieval across languages
  80. Jia, 2021, Scaling up visual and visionlanguage representation learning with noisy text supervision
  81. Jin, 2022, A good prompt is worth millions of parameters? low-resource promptbased learning for vision-language models
  82. Jozefowicz, 2016, Exploring the limits of language modeling
  83. Kamath, 2021, MDETR-modulated detection for end-to-end multimodal understanding
  84. Kay, 2017, The kinetics human action video dataset
  85. Kim, 2018, Bilinear attention networks
  86. Kim, 2016, Multimodal residual learning for visual qa
  87. Kim, 2017, Hadamard product for low-rank bilinear pooling
  88. Kim, 2021, ViLT: Vision-and-language transformer without convolution or region supervision
  89. Kiros, 2014, Unifying visual-semantic embeddings with multimodal neural language models
  90. Ko, 2022, Large-scale bilingual language-image contrastive learning
  91. Kolesnikov, 2022, UViM: A unified modeling approach for vision with learned guiding codes
  92. Krishna, 2017, Dense-captioning events in videos, 10.1109/ICCV.2017.83 · DOI
  93. Krizhevsky, 2012, Imagenet classification with deep convolutional neural networks
  94. Kumar, 2022, Fine-tuning can distort pretrained features and underperform out-of-distribution
  95. Kwon, 2022, Masked vision and language modeling for multi-modal representation learning
  96. Lampert, 2013, Attribute-based classification for zero-shot visual object categorization
  97. Lan, 2020, Albert: A lite bert for self-supervised learning of language representations
  98. Lei, 2022, Revealing single frame bias for video-and-language learning
  99. Lei, 2022, Loopitr: Combining dual and cross encoder architectures for image-text retrieval
  100. Lewis, 2020, Retrieval-augmented generation for knowledge-intensive nlp tasks
  101. Li, 2019, Controllable text-to-image generation
  102. Li, 2022, Language-driven semantic segmentation
  103. Li, 2022, Elevater: A benchmark and toolkit for evaluating language-augmented visual models
  104. Li, 2022, Vision-language intelligence: Tasks, representation learning, and large models
  105. Li, 2022, Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation
  106. Li, 2021, Align before fuse: Vision and language representation learning with momentum distillation
  107. Li, 2022, LAVENDER: Unifying video-language understanding as masked language modeling
  108. Li, 2020, A closer look at the robustness of vision-and-language pre-trained models
  109. Li, 2021, Value: A multi-task benchmark for video-and-language understanding evaluation
  110. Li, 2019, VisualBERT: A simple and performant baseline for vision and language
  111. Li, 2022, Grounded languageimage pre-training
  112. Li, 2022, Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm
  113. Lin, 2004, Rouge: A package for automatic evaluation of summaries
  114. Liu, 2019, VQA challenge 2019 runner-up talk
  115. Liu, 2021, Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing
  116. Liu, 2019, Roberta: A robustly optimized bert pretraining approach
  117. Loshchilov, 2018, Decoupled weight decay regularization
  118. Lu, 2019, Vilbert: Pretraining taskagnostic visiolinguistic representations for vision-and-language tasks
  119. Lu, 2022, Unified-IO: A unified model for vision, language, and multimodal tasks
  120. Lu, 2016, Hierarchical questionimage co-attention for visual question answering
  121. Luo, 2020, Univl: A unified video and language pre-training model for multimodal understanding and generation
  122. Luo, 2021, Clip4clip: An empirical study of clip for end to end video clip retrieval
  123. Mansimov, 2016, Generating images from captions with attention
  124. Mao, 2019, The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision
  125. Mao, 2014, Deep captioning with multimodal recurrent neural networks (m-rnn)
  126. Marino, 2019, Okvqa: A visual question answering benchmark requiring external knowledge
  127. Mikolov, 2013, Efficient estimation of word representations in vector space
  128. Mikolov, 2013, Distributed representations of words and phrases and their compositionality
  129. Minaee, 2021, Deep learning–based text classification: A comprehensive review
  130. Minderer, 2022, Simple open-vocabulary object detection with vision transformers
  131. Mu, 2021, SLIP: Self-supervision meets language-image pre-training
  132. Mustafa, 2022, Multimodal contrastive learning with limoe: The language-image mixture of experts
  133. Nichol, 2021, Glide: Towards photorealistic image generation and editing with text-guided diffusion models
  134. Nie, 2021, Mlp architectures for vision-and-language modeling: An empirical study
  135. Norcliffe-Brown, 2018, Learning conditioned graph structures for interpretable visual question answering
  136. van den Oord, 2017, Neural discrete representation learning
  137. Ordonez, 2011, Im2text: Describing images using 1 million captioned photographs
  138. Pan, 2020, Auto-captions on GIF: A large-scale video-sentence dataset for vision-language pre-training
  139. Papineni, 2002, Bleu: A method for automatic evaluation of machine translation
  140. Parcalabescu, 2021, VALSE: A task-independent benchmark for vision and language models centered on linguistic phenomena
  141. Parcalabescu, 2020, Seeing past words: Testing the cross-modal capabilities of pretrained v and l models on counting tasks
  142. Patrick, 2020, Support-set bottlenecks for video-text representation learning
  143. Pham, 2021, Combined scaling for zero-shot transfer learning
  144. Qian, 2022, Multimodal open-vocabulary video classification via pre-trained vision and language models
  145. Radford, 2021, Learning transferable visual models from natural language supervision
  146. Raffel, 2020, Exploring the limits of transfer learning with a unified text-to-text transformer
  147. Ramesh, 2022, Hierarchical text-conditional image generation with clip latents
  148. Ramesh, 2021, Zero-shot text-to-image generation
  149. Rao, 2021, Denseclip: Language-guided dense prediction with context-aware prompting
  150. Razavi, 2019, Generating diverse high-fidelity images with vq-vae-2
  151. Redmon, 2018, Yolov3: An incremental improvement
  152. Reed, 2016, Generative adversarial text to image synthesis
  153. Reed, 2022, A generalist agent
  154. Ren, 2015, Exploring models and data for image question answering
  155. Ren, 2015, Faster r-cnn: Towards real-time object detection with region proposal networks
  156. Rouditchenko, 2021, Avlnet: Learning audio-visual language representations from instructional videos
  157. Saharia, 2022, Photorealistic text-to-image diffusion models with deep language understanding
  158. Saito, 2022, Prefix conditioning unifies language and label supervision
  159. Santoro, 2017, A simple neural network module for relational reasoning
  160. Schuhmann, 2021, LAION-400M: Open dataset of clip-filtered 400 million imagetext pairs
  161. Sennrich, 2016, Neural machine translation of rare words with subword units, 10.18653/v1/P16-1162 · DOI · Paper page
  162. Shazeer, 2017, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
  163. Shen, 2022, K-lite: Learning transferable visual models with external knowledge
  164. Shen, 2022, How much can clip benefit vision-and- language tasks?
  165. Sheng, 2021, Human-adversarial visual question answering
  166. Shevchenko, 2021, Reasoning over vision and language: E xploring the benefits of supplemental knowledge
  167. Shoeybi, 2019, Megatron-lm: Training multi-billion parameter language models using model parallelism
  168. Shonenkov, 2022, RuCLIP–new models and experiments: A technical report
  169. Simonyan, 2014, Very deep convolutional networks for large-scale image recognition
  170. Singer, 2022, Make-a-video: Text-to-video generation without text-video data
  171. Song, 2021, Denoising diffusion implicit models
  172. Su, 2019, VL-BERT: Pre-training of generic visual-linguistic representations
  173. Su, 2022, Language models can see: Plugging visual controls in text generation
  174. Sung, 2022, Lst: Ladder side-tuning for parameter and memory efficient transfer learning
  175. Sutskever, 2014, Sequence to sequence learning with neural networks
  176. Tan, 2021, VIMPAC: Video pretraining via masked token prediction and contrastive learning
  177. Tang, 2021, Vidlankd: Improving language understanding via video-distilled knowledge transfer
  178. Tian, 2021, Vl-ltr: Learning class-wise visual-linguistic representation for long-tailed visual recognition
  179. Torabi, 2016, Learning language-visual embedding for movie understanding with natural-language
  180. Touvron, 2021, Training data-efficient image transformers and distillation through attention
  181. Tsimpoukelli, 2021, Multimodal few-shot learning with frozen language models
  182. Vaswani, 2017, Attention is all you need
  183. Vedantam, 2019, Probabilistic neural symbolic models for interpretable visual question answering
  184. Villegas, 2022, Phenaki: Variable length video generation from open domain textual description
  185. Wang, 2022, All in one: Exploring unified video-language pre-training
  186. Wang, 2019, Superglue: A stickier benchmark for general-purpose language understanding systems
  187. Wang, 2019, GLUE: A multi-task benchmark and analysis platform for natural language understanding
  188. Wang, 2021, UFO: A unified transformer for vision-language representation learning
  189. Wang, 2020, Minivlm: A smaller and faster vision-language model
  190. Wang, 2022, GIT: A generative image-to-text transformer for vision and language
  191. Wang, 2022, Omnivl: One foundation model for image-language and video-language tasks
  192. Wang, 2018, Learning two-branch neural networks for image-text matching tasks
  193. Wang, 2016, Learning deep structurepreserving image-text embeddings
  194. Wang, 2021, Actionclip: A new paradigm for video action recognition
  195. Wang, 2017, Fvqa: Fact-based visual question answering
  196. Wang, 2022, OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework
  197. Wang, 2022, Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework
  198. Wang, 2022, Visually-augmented language modeling
  199. Wang, 2022, Image as a foreign language: BEiT pretraining for all vision and vision-language tasks
  200. Wang, 2021, VLMo: Unified visionlanguage pre-training with mixture-of-modality-experts

200 of 242 shown

Cited by

Sorted by citations

See all 155

Related

See all

Metadata

Authors
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J.
Affiliations
Microsoft (United States) (US)
ORCID
Venue
Foundations and Trends® in Computer Graphics and Vision
Publisher
Emerald
Published
2022-12-05
Volume
14
Issue
3-4
Pages
163-352
ISSN
  • 1572-2740 (ISSN-L)
  • 1572-2759
DOI
10.1561/0600000105
OpenAlex
W4312810944
Cited by
155
References
291
Citations by year
2026: 20 · 2025: 54 · 2024: 49 · 2023: 32
FWCI
11.05
Type
journal article
Language
en
Open access
Locations
Foundations and Trends® in Computer Graphics and Vision (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
vision-language pre-training, multimodal learning, image captioning, image-text retrieval, visual question answering, visual grounding, video captioning, video-text retrieval, video question answering, image classification, object detection, semantic segmentation, foundation models, unified modeling, robustness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review