The DOI matches its OpenAlex record; 38 of the 38 references it cites were found in a registry or an index.
Unifying Vision-and-Language Tasks via Text Generation
Open accessAbstract
Existing methods for vision-and-language learning typically require designing task-specific architectures and objectives for each task. For example, a multi-label answer classifier for visual question answering, a region scorer for referring expression comprehension, and a language decoder for image captioning, etc. To alleviate these hassles, in this work, we propose a unified framework that learns different tasks in a single architecture with the same language modeling objective, i.e., multimodal conditional text generation, where our models learn to generate labels in text based on the visual and textual inputs. On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-language models. Moreover, our generative approach shows better generalization ability on questions that have rare answers. Also, we show that our framework allows multi-task learning in a single architecture with a single set of parameters, achieving similar performance to separately optimized single-task models. Our code is publicly available at: https://github.com/j-min/VL-T5
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Besides the homogenization of approaches, we also see the homogenization of actual models across research communities in the form of multimodal models”
From On the Opportunities and Risks of Foundation Models (2021) · cites this paper as Cho et al. 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Research works printed but not linked: 39
- Anderson, P., Fernando, B., Johnson, M., and Gould, S. SPICE: Semantic Propositional Image Caption Evalua- tion. In ECCV, 2016.
- Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question An- swering. In CVPR, 2018. URL http://arxiv.org/ abs/1707.07998. · arXiv
- Banerjee, S. and Lavie, A. METEOR : An Automatic Met- ric for MT Evaluation with Improved Correlation with Human Judgments. In ACL Workshop, 2005.
- Caglayan, O., Madhyastha, P., Specia, L., and Barrault, L. Probing the Need for Visual Context in Multi- modal Machine Translation. In NAACL, 2019. ISBN 9781950737130. doi: 10.18653/v1/n19-1422. · DOI
- Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. Elec- tra: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020.
- Elliott, D., Frank, S., Sima'an, K., and Specia, L. Multi30K : Multilingual English-German Image Descriptions. In ACL Workshop, pp. 70-74, 2016.
- Grönroos, S.-A., Huet, B., Kurimo, M., Laaksonen, J., Meri- aldo, B., Pham, P., Sjöberg, M., Sulubacak, U., Tiede- mann, J., Troncy, R., and Vázquez, R. The MeMAD Submission to the WMT18 Multimodal Translation Task. In WMT, volume 2, pp. 609-617, 2018.
- He, K., Gkioxari, G., Dollar, P., and Girshick, R. Mask R-CNN. ICCV, 2017.
- Huang, Z., Zeng, Z., Liu, B., Fu, D., and Fu, J. Pixel- BERT: Aligning Image Pixels with Text by Deep Multi- Modal Transformers. 2020. URL http://arxiv. org/abs/2004.00849.
- Hudson, D. A. and Manning, C. D. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. ISBN 9781728132938. doi: 10.1109/CVPR.2019.00686. · DOI · Paper page
- Keskar, N. S., McCann, B., Xiong, C., and Socher, R. Unify- ing Question Answering and Text Classification via Span Extraction. 2019. URL http://arxiv.org/abs/ 1904.09286.
- Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., and Hajishirzi, H. Unified QA : Crossing Format Boundaries with a Single QA System. In Findings of EMNLP, 2020.
- Kim, J.-h., Jun, J., and Zhang, B.-t. Bilinear Attention Networks. In NeurIPS, pp. 1-12, 2018.
- Li, L., Chen, Y.-C., Yu Cheng, Z. G., Yu, L., and Liu, J. HERO: Hierarchical Encoder for Video+Language Omni- representation Pre-training. In EMNLP, 2020a.
- Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y., and Gao, J. Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. In ECCV, 2020b. URL http://arxiv.org/abs/2004.06165. · arXiv
- Loshchilov, I. and Hutter, F. Decoupled Weight De- cay Regularization. In ICLR, 2019. URL https: //openreview.net/forum?id=Bkg6RiCqY7.
- Lu, J., Goswami, V., Rohrbach, M., Parikh, D., and Lee, S. 12-in-1: Multi-Task Vision and Language Repre- sentation Learning. In CVPR, 2020. URL http: //arxiv.org/abs/1912.02315.
- Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., and Murphy, K. Generation and Comprehension of Unam- biguous Object Descriptions. In CVPR, 2016.
- Miech, A., Alayrac, J.-B., Smaira, L., Laptev, I., Sivic, J., and Zisserman, A. End-to-end learning of visual repre- sentations from uncurated instructional videos. In CVPR, 2020.
- Nogueira, R., Jiang, Z., Lin, J., Mar, I. R., Pradeep, R., and Lin, J. Document Ranking with a Pretrained Sequence- to-Sequence Model. In Findings of EMNLP, pp. 1-8, 2020.
- Paszke, A., Gross, S., Chintala, S., Chana, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in PyTorch. In NIPS Workshop, 2017. URL https://openreview. net/pdf?id=BJJsrmfCZ.
- Press, O. and Wolf, L. Using the Output Embedding to Improve Language Models. In EACL, 2017.
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the Limits of Transfer Learning with a Unified Text-to- Text Transformer. JMLR, 21:1-67, 2020. URL http: //arxiv.org/abs/1910.10683.
- Ren, S., He, K., Girshick, R., and Sun, J. Faster R- CNN: Towards Real-Time Object Detection with Re- gion Proposal Networks. In NIPS, 2015. URL https: //arxiv.org/abs/1506.01497.
- Shin, T., Razeghi, Y., IV, R. L. L., Wallace, E., and Singh, S. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP, 2020.
- Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. MASS: Masked Sequence to Sequence Pre-training for Language Generation. In ICML, 2019. URL http://arxiv. org/abs/1905.02450.
- Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., and Artzi, Y. A Corpus for Reasoning About Natural Language Grounded in Photographs. In ACL, 2019. URL http: //arxiv.org/abs/1811.00491.
- Sun, C., Baradel, F., Murphy, K., and Schmid, C. Con- trastive Bidirectional Transformer for Temporal Represen- tation Learning. 2019a. URL http://arxiv.org/ abs/1906.05743. · arXiv
- Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. VideoBERT: A Joint Model for Video and Language Representation Learning. In ICCV, 2019b. URL http: //arxiv.org/abs/1904.01766.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polo- sukhin, I. Attention Is All You Need. In NIPS, 2017. URL https://papers.nips.cc/paper/ 7181-attention-is-all-you-need.pdf.
- Vedantam, R., Zitnick, C. L., and Parikh, D. CIDEr: Consensus-based Image Description Evaluation. In CVPR, nov 2015. URL http://arxiv.org/abs/ 1411.5726.
- Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analy- sis platform for natural language understanding. In ICLR, 2018.
- Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. HuggingFace's Transformers: State- of-the-art Natural Language Processing. 2019. URL http://arxiv.org/abs/1910.03771. · arXiv
- Xu, J., Mei, T., Yao, T., and Rui, Y. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
- Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., and Berg, T. L. MAttNet : Modular Attention Network for Referring Expression Comprehension. In CVPR, 2018a. URL https://arxiv.org/abs/1801.08186. · arXiv
- Yu, Y., Kim, J., and Kim, G. A joint sequence fusion model for video question answering and retrieval. In ECCV, 2018b.
- Zellers, R., Bisk, Y., Schwartz, R., and Choi, Y. Swag: A large-scale adversarial dataset for grounded common- sense inference. In EMNLP, 2018.
- Zhou, L., Xu, C., and Corso, J. J. Towards automatic learn- ing of procedures from web instructional videos. In AAAI, 2018.
- Zhu, L. and Yang, Y. Actbert: Learning global-local video- text representations. In CVPR, 2020.
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Li, J., Li, D., Xiong, C., & Hoi, S. (2022) · arXiv · doi:10.48550/arxiv.2201.12086 · Cited by 1,741
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
“We show in row (ii) the importance of our gradient accumulation strategy compared to using round-robin updates [17].”
BackgroundMethodResultInfluentialMDETR - Modulated Detection for End-to-End Multi-Modal Understanding
Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., & Carion, N. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00180 · Cited by 729
Metadata
- Authors
- Cho, J., Lei, J., Tan, H. X., & Bansal, M.
- Venue
- arXiv (Cornell University)
- Publisher
- arXiv
- Published
- 2021
- Pages
- 1931-1942
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.02779
- OpenAlex
- W3126464137
- Cited by
- 179
- References
- 38
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Unifying Vision-and-Language Tasks via Text Generation. https://app.cochat.ai/papers/10.48550/arxiv.2102.02779
- The paper
- Cho, J., Lei, J., Tan, H. X., & Bansal, M. (2021). Unifying Vision-and-Language Tasks via Text Generation. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2102.02779
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.