CoChat
Sign Up
PreprintCited by 2,109Multimodal Machine Learning ApplicationsVisual Attention and Saliency DetectionTopic Modeling

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Open access
Lu, Jiasen, Batra, Dhruv, Parikh, Devi, Lee, Stefan·2019·arXiv
DOI 10.48550/arxiv.1908.02265arXiv 1908.02265
Open this paper in CoChat — free accountOpen PDF

Abstract

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1908.02265

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Vision-language models combine a language model with an image encoder and are trained on text-image pairs, enabling prediction of word continuations in the context of an image.

    “vision-language models (VLMs) such as VilBERT 19 and Flamingo 2 combine a language model with an image encoder and are trained on a multi-modal corpus of text-image pairs.This enables them to predict how a given sequence of words will continue in the context of a given image.”

    From Talking about Large Language Models (2024) · cites this paper as [19]

    The Introduction (p. 2) and Section 2.2 (p. 3) describe ViLBERT’s visual and language streams and training on paired images and text. But Section 2.2 describes reconstructing masked words, not predicting word continuations; Related Work (p. 8) says decoding output text from this bidirectional model remains unclear. The continuation-prediction part is therefore not supported for ViLBERT. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • Existing solutions in multimodal Transformer adopt a fixed fusion strategy.

    “Existing solutions in multimodal Transformer adopt a fixed fusion strategy [18,21,24,26,33].”

    From Are Multimodal Transformers Robust to Missing Modality? (2022) · cites this paper as [26]

  • Harmonic CNN uses trainable filters to model harmonic structures in the audio.

    “and Harmonic CNN [30], which uses trainable filters to model harmonic structures in the audio.”

    From MusCaps: Generating Captions for Music Audio (2021) · cites this paper as [30]

    The Introduction (p. 2) describes ViLBERT as a model with visual and linguistic streams, while Training ViLBERT (p. 5) describes extracting image-region features. Searches for Harmonic CNN, trainable filters, and harmonic audio structures found no passage addressing the claim. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 30 · Cited by 2,109

Research works printed but not linked: 38

  1. [1][1] Margaret A. Boden. Mind as Machine: A History of Cognitive Science . Oxford University Press, 2008. -
  2. [2][2] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. Referit game: Referring to objects in photographs of natural scenes. In EMNLP , 2014. -
  3. [3][3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In ICCV , 2015. -
  4. [4][4] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR , 2017. -
  5. [5][5] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR , abs/1504.00325, 2015. - · arXiv
  6. [6][6] Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurelie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. "foil it! find one mismatch between image and language caption". In ACL , 2017. -
  7. [7][7] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied Question Answering. In CVPR , 2018. -
  8. [8][8] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR) , 2018. -
  9. [9][9] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In CVPR , 2018. -
  10. [10][10] Harsh Agrawal, Karan Desai, Xinlei Chen, Rishabh Jain, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. arXiv preprint arXiv:1812.08658 , 2018. - · arXiv
  11. [16][16] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In arXiv , 2016. URL https://arxiv.org/abs/1602.07332 . - · arXiv
  12. [17][17] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV , 2015. -
  13. [19][19] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. In arXiv , 2014. -
  14. [20][20] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In CVPR , pages 6874-6883, 2017. -
  15. [21][21] Dinesh Jayaraman, Ruohan Gao, and Kristen Grauman. Shapecodes: self-supervised feature learning by lifting views to viewgrids. In ECCV , pages 120-136, 2018. -
  16. [22][22] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV , pages 609-617, 2017. -
  17. [23][23] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In CVPR , pages 2701-2710, 2017. -
  18. [24][24] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL , 2018. -
  19. [25][25] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In CVPR , 2019. -
  20. [26][26] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL , 2014. -
  21. [27][27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017. -
  22. [28][28] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 , 2016. - · arXiv
  23. [29][29] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. arXiv preprint arXiv:1904.01766 , 2019. - · arXiv
  24. [30][30] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR , pages 6077-6086, 2018. -
  25. [31][31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NuerIPS , pages 91-99, 2015. -
  26. [32][32] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP , 2014. -
  27. [33][33] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In CVPR , 2018. -
  28. [34][34] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV , pages 2961-2969, 2017. -
  29. [35][35] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV , pages 201-216, 2018. -
  30. [36][36] Gao Peng, Hongsheng Li, Haoxuan You, Zhengkai Jiang, Pan Lu, Steven Hoi, and Xiaogang Wang. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. arXiv preprint arXiv:1812.05252 , 2018. - · arXiv
  31. [37][37] Alex Wang and Kyunghyun Cho. Bert has a mouth, and it must speak: Bert as a markov random field language model. arXiv preprint arXiv:1902.04094 , 2019. - · arXiv
  32. [38][38] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV , pages 1422-1430, 2015. -
  33. [39][39] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV , pages 649-666. Springer, 2016. -
  34. [40][40] Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE PAMI , 38(9):1734-1747, 2015. -
  35. [41][41] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR , pages 2536-2544, 2016. -
  36. [42][42] Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In CVPR , pages 1413-1421, 2015. -
  37. [43][43] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV , pages 527-544. Springer, 2016. -
  38. [44][44] Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291 , 2019. - · arXiv

Authors
Lu, J., Batra, D., Parikh, D., & Lee, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1908.02265
arXiv
1908.02265
OpenAlex
W2966715458
Cited by
2,109
References
30
Citations by year
2026: 8 · 2025: 239 · 2024: 444 · 2023: 575 · 2022: 435 · 2021: 316 · 2020: 90 · 2019: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
multimodal transformer, co-attention, visual question answering, visual commonsense reasoning, referring expressions, visual grounding, transfer learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review