Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
PreprintCited by 179

The DOI matches its OpenAlex record; 38 of the 38 references it cites were found in a registry or an index.

Unifying Vision-and-Language Tasks via Text Generation

Open access
Jaemin Cho, Lei, Jie, Hao Xuan Tan, Mohit Bansal · 2021 · arXiv (Cornell University)

Abstract

Existing methods for vision-and-language learning typically require designing task-specific architectures and objectives for each task. For example, a multi-label answer classifier for visual question answering, a region scorer for referring expression comprehension, and a language decoder for image captioning, etc. To alleviate these hassles, in this work, we propose a unified framework that learns different tasks in a single architecture with the same language modeling objective, i.e., multimodal conditional text generation, where our models learn to generate labels in text based on the visual and textual inputs. On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-language models. Moreover, our generative approach shows better generalization ability on questions that have rare answers. Also, we show that our framework allows multi-task learning in a single architecture with a single set of parameters, achieving similar performance to separately optimized single-task models. Our code is publicly available at: https://github.com/j-min/VL-T5

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

OpenAlex · 10.48550/arxiv.2102.02779

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 38 references

Research works printed but not linked: 39

  1. Anderson, P., Fernando, B., Johnson, M., and Gould, S. SPICE: Semantic Propositional Image Caption Evalua- tion. In ECCV, 2016.
  2. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question An- swering. In CVPR, 2018. URL http://arxiv.org/ abs/1707.07998. · arXiv
  3. Banerjee, S. and Lavie, A. METEOR : An Automatic Met- ric for MT Evaluation with Improved Correlation with Human Judgments. In ACL Workshop, 2005.
  4. Caglayan, O., Madhyastha, P., Specia, L., and Barrault, L. Probing the Need for Visual Context in Multi- modal Machine Translation. In NAACL, 2019. ISBN 9781950737130. doi: 10.18653/v1/n19-1422. · DOI
  5. Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. Elec- tra: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020.
  6. Elliott, D., Frank, S., Sima'an, K., and Specia, L. Multi30K : Multilingual English-German Image Descriptions. In ACL Workshop, pp. 70-74, 2016.
  7. Grönroos, S.-A., Huet, B., Kurimo, M., Laaksonen, J., Meri- aldo, B., Pham, P., Sjöberg, M., Sulubacak, U., Tiede- mann, J., Troncy, R., and Vázquez, R. The MeMAD Submission to the WMT18 Multimodal Translation Task. In WMT, volume 2, pp. 609-617, 2018.
  8. He, K., Gkioxari, G., Dollar, P., and Girshick, R. Mask R-CNN. ICCV, 2017.
  9. Huang, Z., Zeng, Z., Liu, B., Fu, D., and Fu, J. Pixel- BERT: Aligning Image Pixels with Text by Deep Multi- Modal Transformers. 2020. URL http://arxiv. org/abs/2004.00849.
  10. Hudson, D. A. and Manning, C. D. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. ISBN 9781728132938. doi: 10.1109/CVPR.2019.00686. · DOI · Paper page
  11. Keskar, N. S., McCann, B., Xiong, C., and Socher, R. Unify- ing Question Answering and Text Classification via Span Extraction. 2019. URL http://arxiv.org/abs/ 1904.09286.
  12. Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., and Hajishirzi, H. Unified QA : Crossing Format Boundaries with a Single QA System. In Findings of EMNLP, 2020.
  13. Kim, J.-h., Jun, J., and Zhang, B.-t. Bilinear Attention Networks. In NeurIPS, pp. 1-12, 2018.
  14. Li, L., Chen, Y.-C., Yu Cheng, Z. G., Yu, L., and Liu, J. HERO: Hierarchical Encoder for Video+Language Omni- representation Pre-training. In EMNLP, 2020a.
  15. Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y., and Gao, J. Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. In ECCV, 2020b. URL http://arxiv.org/abs/2004.06165. · arXiv
  16. Loshchilov, I. and Hutter, F. Decoupled Weight De- cay Regularization. In ICLR, 2019. URL https: //openreview.net/forum?id=Bkg6RiCqY7.
  17. Lu, J., Goswami, V., Rohrbach, M., Parikh, D., and Lee, S. 12-in-1: Multi-Task Vision and Language Repre- sentation Learning. In CVPR, 2020. URL http: //arxiv.org/abs/1912.02315.
  18. Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., and Murphy, K. Generation and Comprehension of Unam- biguous Object Descriptions. In CVPR, 2016.
  19. Miech, A., Alayrac, J.-B., Smaira, L., Laptev, I., Sivic, J., and Zisserman, A. End-to-end learning of visual repre- sentations from uncurated instructional videos. In CVPR, 2020.
  20. Nogueira, R., Jiang, Z., Lin, J., Mar, I. R., Pradeep, R., and Lin, J. Document Ranking with a Pretrained Sequence- to-Sequence Model. In Findings of EMNLP, pp. 1-8, 2020.
  21. Paszke, A., Gross, S., Chintala, S., Chana, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in PyTorch. In NIPS Workshop, 2017. URL https://openreview. net/pdf?id=BJJsrmfCZ.
  22. Press, O. and Wolf, L. Using the Output Embedding to Improve Language Models. In EACL, 2017.
  23. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the Limits of Transfer Learning with a Unified Text-to- Text Transformer. JMLR, 21:1-67, 2020. URL http: //arxiv.org/abs/1910.10683.
  24. Ren, S., He, K., Girshick, R., and Sun, J. Faster R- CNN: Towards Real-Time Object Detection with Re- gion Proposal Networks. In NIPS, 2015. URL https: //arxiv.org/abs/1506.01497.
  25. Shin, T., Razeghi, Y., IV, R. L. L., Wallace, E., and Singh, S. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP, 2020.
  26. Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. MASS: Masked Sequence to Sequence Pre-training for Language Generation. In ICML, 2019. URL http://arxiv. org/abs/1905.02450.
  27. Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., and Artzi, Y. A Corpus for Reasoning About Natural Language Grounded in Photographs. In ACL, 2019. URL http: //arxiv.org/abs/1811.00491.
  28. Sun, C., Baradel, F., Murphy, K., and Schmid, C. Con- trastive Bidirectional Transformer for Temporal Represen- tation Learning. 2019a. URL http://arxiv.org/ abs/1906.05743. · arXiv
  29. Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. VideoBERT: A Joint Model for Video and Language Representation Learning. In ICCV, 2019b. URL http: //arxiv.org/abs/1904.01766.
  30. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polo- sukhin, I. Attention Is All You Need. In NIPS, 2017. URL https://papers.nips.cc/paper/ 7181-attention-is-all-you-need.pdf.
  31. Vedantam, R., Zitnick, C. L., and Parikh, D. CIDEr: Consensus-based Image Description Evaluation. In CVPR, nov 2015. URL http://arxiv.org/abs/ 1411.5726.
  32. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analy- sis platform for natural language understanding. In ICLR, 2018.
  33. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. HuggingFace's Transformers: State- of-the-art Natural Language Processing. 2019. URL http://arxiv.org/abs/1910.03771. · arXiv
  34. Xu, J., Mei, T., Yao, T., and Rui, Y. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
  35. Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., and Berg, T. L. MAttNet : Modular Attention Network for Referring Expression Comprehension. In CVPR, 2018a. URL https://arxiv.org/abs/1801.08186. · arXiv
  36. Yu, Y., Kim, J., and Kim, G. A joint sequence fusion model for video question answering and retrieval. In ECCV, 2018b.
  37. Zellers, R., Bisk, Y., Schwartz, R., and Choi, Y. Swag: A large-scale adversarial dataset for grounded common- sense inference. In EMNLP, 2018.
  38. Zhou, L., Xu, C., and Corso, J. J. Towards automatic learn- ing of procedures from web instructional videos. In AAAI, 2018.
  39. Zhu, L. and Yang, Y. Actbert: Learning global-local video- text representations. In CVPR, 2020.

Cited by

See all 179

Metadata

Authors
Cho, J., Lei, J., Tan, H. X., & Bansal, M.
Venue
arXiv (Cornell University)
Publisher
arXiv
Published
2021
Pages
1931-1942
ISSN
2331-8422 (ISSN-L)
OpenAlex
W3126464137
Cited by
179
References
38
Type
posted content
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Unifying Vision-and-Language Tasks via Text Generation. https://app.cochat.ai/papers/10.48550/arxiv.2102.02779
The paper
Cho, J., Lei, J., Tan, H. X., & Bansal, M. (2021). Unifying Vision-and-Language Tasks via Text Generation. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2102.02779

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.