UNITER: UNiversal Image-TExt Representation Learning
Open accessIntegrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Transformer architectures have emerged as a popular choice due to their suitability for both language and image data.
“Transformer architectures have emerged as a popular choice due to their suitability for both language and image data [27, 73]”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [27]
Tasks are more difficult; more abstract multimodal tasks are proposed.
“Tasks are more difficult.Beyond the straightforward tasks, more abstract multimodal tasks are proposed, e.g., MultiMET [153] (a multimodal dataset for metaphor understanding), Hateful Memes [154] (hate speech in multimodal memes).”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [153]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
Self-citationVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
Zhang, R., Isola, P., & Efros, A. A. (2016) · European Conference on Computer Vision · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,803 · see citing works
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · see citing works
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · see citing works
Unsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477 · see citing works
Computational Optimal Transport with Applications to Data Sciences
Gabriel Peyré, A., & Cuturi, M. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000073 · Cited by 2,354 · see citing works
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · see citing works
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · see citing works
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · see citing works
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444 · see citing works
Stacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402 · see citing works
VideoBERT: A Joint Model for Video and Language Representation Learning
Sun, C., Myers, A., Vondrick, C., Murphy, K. P., & Schmid, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,212 · see citing works
ReferItGame: Referring to Objects in Photographs of Natural Scenes
Kazemzadeh, S., Ordóñez, V., Matten, M., & Berg, T. L. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1086 · Cited by 1,082 · see citing works
(2016) · doi:10.1007/978-3-319-46475-6_5 · Cited by 1,081 · see citing works
Deep Modular Co-Attention Networks for Visual Question Answering
Zhou, Y., Yu, J., Cui, Y., Tao, D., & Tian, Q. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00644 · Cited by 1,013 · see citing works
VQA: Visual Question AnsweringPreprint
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Batra, D., & Parikh, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00468 · Cited by 968 · see citing works
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950 · see citing works
Computational Optimal Transport
Gabriel Peyré, A., & Cuturi, M. (2019) · now publishers, Inc. eBooks · doi:10.1561/9781680835519 · Cited by 909 · see citing works
Im2Text: Describing Images Using 1 Million Captioned Photographs
Ordóñez, V., Kulkarni, G. A., & Berg, T. L. (2011) · Cited by 879 · see citing works
Unified Vision-Language Pre-Training for Image Captioning and VQA
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7005 · Cited by 864 · see citing works
MAttNet: Modular Attention Network for Referring Expression Comprehension
Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., & Berg, T. L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00142 · Cited by 857 · see citing works
Self-citationUnicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
Li, G., Duan, N., Fang, Y., Gong, M., & Jiang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6795 · Cited by 795 · see citing works
Learning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.541 · Cited by 788 · see citing works
From Recognition to Cognition: Visual Commonsense Reasoning
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00688 · Cited by 769 · see citing works
Revealing the Dark Secrets of BERT
Kovaleva, O., Romanov, A., Rogers, A., & Rumshisky, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1445 · Cited by 537 · see citing works
A Corpus for Reasoning about Natural Language Grounded in Photographs
Suhr, A., Zhou, S., Zhang, A. Q., Zhang, I., Bai, H., & Artzi, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1644 · Cited by 487 · see citing works
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering
Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S. C. H., Wang, X., & Li, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00680 · Cited by 415 · see citing works
12-in-1: Multi-Task Vision and Language Representation Learning
Lu, J., Goswami, V., Rohrbach, M., Parikh, D., & Lee, S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01045 · Cited by 410 · see citing works
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2020) · International Conference on Learning Representations · Cited by 276 · see citing works
Fusion of Detected Objects in Text for Visual Question Answering
Alberti, C., Ling, J., Collins, M., & Reitter, D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1219 · Cited by 179 · see citing works
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingPreprint
Li, G., Duan, N., Fang, Y., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06066 · Cited by 115 · see citing works
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
Cao, J., Gan, Z., Cheng, Y., Yu, L., Chen, Y.-C., & Liu, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58539-6_34 · Cited by 105 · see citing works
Self-citationA Fast Proximal Point Method for Wasserstein Distance.Preprint
Xie, Y., Wang, X., Wang, R., & Hongyuan Zha, B. (2018) · arXiv (Cornell University) · Cited by 30 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Cited by
Sorted by citations
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
“…for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classification (Wu et al., 2020), unsupervised object discovery (Locatello et al., 2020), or unified text-vision tasks (Chen et al., 2020c; Lu et al., 2019; Li et al., 2019).”
MethodScaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · see citing works
“This is one of the most effective loss functions for both self-supervised (Chen et al., 2020b) and supervised (Zhai & Wu, 2019; Musgrave et al., 2020) representation learning.”
BackgroundMethodInfluentialSelf-citationAlign before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078 · see citing works
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994 · see citing works
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BackgroundImage as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks
Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01838 · Cited by 561 · see citing works
Zhao, F., Zhang, C., & Geng, B. (2024) · ACM Computing Surveys · doi:10.1145/3649447 · Cited by 480 · see citing works
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429 · see citing works
“LXMERT [64], UNITER [10], VinVL [80], Oscar [34], VilBert [45], and VLP [86] are among the earliest work to train universal vision-language models to tackle many multimodal tasks.”
BackgroundUnderstanding Robustness of Transformers for Image Classification
Bhojanapalli, S., Chakrabarti, A., Glasner, D., Li, D., Unterthiner, T., & Veit, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01007 · Cited by 440 · see citing works
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
Li, L., Chen, Y.-C., Cheng, Y., Gan, Z., Yu, L., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.161 · Cited by 408 · see citing works
Self-citationLarge language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395 · see citing works
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing
Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20059-5_1 · Cited by 317 · see citing works
“A variety of self-supervised VLP approaches have been proposed towards jointly learning visual and textual representations of paired data without supervision, such as frameworks using contrastive objectives [26, 42, 60], approaches based on joint transformer architectures [40, 41, 51, 69], self-supervised VLP with wor…”
BackgroundInfluentialFoundation Models Defining a New Era in Vision: A Survey and Outlook
Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, F. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3506283 · Cited by 304 · see citing works
Vision-Language Pre-Training with Triple Contrastive Learning
Yang, J., Duan, J., Tran, S., Xu, Y., Chanda, S., Chen, L., Zeng, B., Chilimbi, T., & Huang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01522 · Cited by 309 · see citing works
UniT: Multimodal Multitask Learning with a Unified Transformer
Hu, R., & Singh, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00147 · Cited by 303 · see citing works
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation
Hong, Y., Wu, Q., Qi, Y., Rodriguez-Opazo, C., & Gould, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00169 · Cited by 264 · see citing works
MultiMAE: Multi-modal Multi-task Masked Autoencoders
Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_20 · Cited by 268 · see citing works
Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models
Liu, Z., Rodriguez-Opazo, C., Teney, D., & Gould, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00213 · Cited by 238 · see citing works
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
Bannur, S., Hyland, S., Liu, Q., Pérez-García, F., Ilse, M., Castro, D. C., Boecking, B., Sharma, H., Bouzid, K., Thieme, A., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01442 · Cited by 227 · see citing works
CLIPort: What and Where Pathways for Robotic Manipulation
Shridhar, M., Manuelli, L., & Fox, D. (2021) · arXiv · doi:10.48550/arxiv.2109.12098 · Cited by 223 · see citing works
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works
Are Multimodal Transformers Robust to Missing Modality?
Ma, M., Ren, J., Zhao, L., Testuggine, D., & Peng, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01764 · Cited by 198 · see citing works
“So far, research on Trans-former models has been limited to developing new architectures for fusion [29, 35, 38] or exploring better self-supervised learning tasks [1, 6, 8, 47, 48].”
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182 · see citing works
Causal Attention for Vision-Language Tasks
Yang, X., Zhang, H., Qi, G., & Cai, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00972 · Cited by 175 · see citing works
Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155 · see citing works
Self-citationAirbert: In-domain Pretraining for Vision-and-Language Navigation
Guhur, P.-L., Tapaswi, M., Chen, S., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00166 · Cited by 139 · see citing works
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
Yang, J., Bisk, Y., & Gao, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01136 · Cited by 115 · see citing works
CLIP-Event: Connecting Text and Images with Event Structures
Li, M., Xu, R., Wang, S., Zhou, L., Lin, X., Zhu, C., Zeng, M., Ji, H., & Chang, S.-F. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01593 · Cited by 117 · see citing works
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
Hendricks, L. A., Mellor, J., Schneider, R., Alayrac, J.-B., & Nematzadeh, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00385 · Cited by 100 · see citing works
Object Detection with Transformers: A Review
Shehzadi, T., Hashmi, K. A., Liwicki, M., Stricker, D., & Afzal, M. Z. (2025) · Sensors · doi:10.3390/s25196025 · Cited by 93 · see citing works
JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs
Ke, P., Ji, H., Ran, Y., Cui, X., Wang, L., Song, L., Zhu, X., & Huang, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.223 · Cited by 92 · see citing works
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
Xu, H., Yan, M., Li, C., Bi, B., Huang, S., Xiao, W., & Huang, F. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.42 · Cited by 84 · see citing works
Video Transformer for Deepfake Detection with Incremental Learning
Khan, S. A., & Dai, H. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475332 · Cited by 79 · see citing works
Qi, Y., Pan, Z., Hong, Y., Yang, M.-H., van den Hengel, A., & Wu, Q. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00168 · Cited by 80 · see citing works
Learning to Generate Scene Graph from Natural Language Supervision
Zhong, Y., Shi, J., Yang, J., Xu, C., & Li, Y. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00184 · Cited by 73 · see citing works
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks
Kayser, M., Camburu, O.-M., Salewski, L., Emde, C., Do, V., Akata, Z., & Lukasiewicz, T. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00128 · Cited by 68 · see citing works
Liu, J., Capurro, D., Nguyen, A., & Verspoor, K. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104466 · Cited by 66 · see citing works
Self-citationProbing Image-Language Transformers for Verb Understanding
Hendricks, L. A., & Nematzadeh, A. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.318 · Cited by 63 · see citing works
VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering
Wang, Y., Yasunaga, M., Ren, H., Wada, S., & Leskovec, J. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01973 · Cited by 68 · see citing works
A survey of transformer-based multimodal pre-trained modals
Han, X., Wang, Y.-T., Feng, J.-L., Deng, C., Chen, Z.-H., Huang, Y.-A., Su, H., Hu, L., & Hu, P.-W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2022.09.136 · Cited by 59 · see citing works
Context-aware Alignment and Mutual Masking for 3D-Language Pre-training
Jin, Z., Hayat, M., Yang, Y., Guo, Y., & Lei, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01057 · Cited by 56 · see citing works
Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-Modal Pretraining
Zhan, X., Wu, Y., Dong, X., Wei, Y., Lu, M., Zhang, Y., Xu, H., & Liang, X. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01157 · Cited by 54 · see citing works
Khan, S. A., & Dang-Nguyen, D.-T. (2024) · IEEE Access · doi:10.1109/access.2023.3348450 · Cited by 53 · see citing works
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
Memobert: Pre-Training Model with Prompt-Based Learning for Multimodal Emotion Recognition
Zhao, J., Li, R., Jin, Q., Wang, X., & Li, H. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746910 · Cited by 39 · see citing works
GilBERT: Generative Vision-Language Pre-Training for Image-Text Retrieval
Hong, W., Ji, K., Liu, J., Wang, J., Chen, J., & Chu, W. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462838 · Cited by 37 · see citing works
Self-citationCOOKIE: Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation
Wen, K., Xia, J., Huang, Y., Li, L., Xu, J., & Shao, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00221 · Cited by 37 · see citing works
Self-citationVisQA: X-raying Vision and Language Reasoning in Transformers
Jaunet, T., Kervadec, C., Vuillemot, R., Antipov, G., Baccouche, M., & Wolf, C. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114683 · Cited by 30 · see citing works
DeAR: Debiasing Vision-Language Models with Additive Residuals
Seth, A., Hemani, M., & Agarwal, C. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00659 · Cited by 34 · see citing works
Pre-training in Medical Data: A Survey
Qiu, Y., Lin, F., Chen, W., & Xu, M. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1382-8 · Cited by 30 · see citing works
Multistain Pretraining for Slide Representation Learning in Pathology
Jaume, G., Vaidya, A., Zhang, A., H. Song, A., J. Chen, R., Sahai, S., Mo, D., Madrigal, E., Phi Le, L., & Mahmood, F. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73414-4_2 · Cited by 27 · see citing works
How Transferable are Reasoning Patterns in VQA?
Kervadec, C., Jaunet, T., Antipov, G., Baccouche, M., Vuillemot, R., & Wolf, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00419 · Cited by 25 · see citing works
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85 · see citing works
“Transformer architectures have emerged as a popular choice due to their suitability for both language and image data [27, 73] and a recent public toolkit was released for incorporating multimodal data on top of text-based Transformers for prediction tasks [60].”
MethodSelf-citationImproving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment
Ji, Z., Shaikh, M. A., Moukheiber, D., Srihari, S. N., Peng, Y., & Gao, M. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87589-3_12 · Cited by 21 · see citing works
Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA
Mo, W., & Liu, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28222 · Cited by 20 · see citing works
Li, Y., Kong, C., Zhao, G., & Zhao, Z. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11337-0 · Cited by 7 · see citing works
Akdemir, K., & Yanardag, P. (2024) · arXiv · doi:10.48550/arxiv.2406.02820
“…of large-scale text-to-image diffusion models (Rombach et al. 2022; Nichol et al. 2021) has enabled the widespread use of image generation models (Chen et al. 2020; Kim, Son, and Kim 2021; Saharia et al. 2022), largely due to the plentiful availability of image-text pair datasets and their…”
BackgroundAi, Q., Zhan, J., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2501.02842
“…models: In terms of aligning multimodal inputs, there are mainly three approaches: • Object Detection-Based Input: This method involves detecting objects within an image, extracting their features and associated spatial information, and then feeding this data into the language model [55, 56].”
- Authors
- Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J.
- Affiliations
- Microsoft (United States) (US)
- Corresponding author
- Chen, Yen-Chun
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2020
- Pages
- 104-120
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W3090449556
- Cited by
- 2,055
- References
- 44
- Citations by year
- 2026: 160 · 2025: 328 · 2024: 384 · 2023: 498 · 2022: 364 · 2021: 261 · 2020: 57 · 2019: 3
- FWCI
- 91.00
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- image-text representation learning, vision-language pre-training, multimodal embeddings, image-text matching, optimal transport, visual question answering, image-text retrieval, referring expression comprehension, visual commonsense reasoning, visual entailment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
