The DOI matches its DataCite record; 46 of the 46 references it cites were found in a registry or an index.
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Open accessAbstract
We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark. Code is released at \url{https://github.com/jackroos/VL-BERT}.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Vision Transformers are commonly evaluated using the sliding window procedure from MMSegmentation [59] (e.g., [10, 101, 103]).”
From MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022) · cites this paper as [59]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Flagged first, then by citations
Research works printed but not linked: 4
- Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , pp. 1440-1448, 2015.
- Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems , pp. 3294-3302, 2015.
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pp. 1097-1105, 2012.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 1(8), 2019.
Cited by
Sorted by citations
Items 101–200 of 957
Multi-modal Masked Autoencoders for Medical Vision-and-Language Pre-training
Chen, Z., Du, Y., Hu, J., Liu, Y., Li, G., Wan, X., & Chang, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_65 · Cited by 166
Learning Video Representations from Large Language Models
Zhao, Y., Misra, I., Krähenbühl, P., & Girdhar, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00637 · Cited by 190
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text DataPreprint
Qi, D., Su, L., Song, J. L., Cui, E., Bharti, T., & Sacheti, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.07966 · Cited by 255
M3DETR: Multi-representation, Multi-scale, Mutual-relation 3D Object Detection with Transformers
Guan, T., Wang, J., Lan, S., Chandra, R., Wu, Z., Davis, L. S., & Manocha, D. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00235 · Cited by 157
Hierarchical Interactive Multimodal Transformer for Aspect-Based Multimodal Sentiment Analysis
Yu, J., Chen, K., & Xia, R. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3171091 · Cited by 153
TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval
Liu, Y., Xiong, P., Xu, L., Cao, S., & Jin, Q. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19781-9_19 · Cited by 164
SignBERT+: Hand-Model-Aware Self-Supervised Pre-Training for Sign Language Understanding
Hu, H., Zhao, W., Zhou, W., & Li, H. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3269220 · Cited by 167
AOBERT: All-modalities-in-One BERT for multimodal sentiment analysis
Kim, K.-H., & Park, S. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.022 · Cited by 149
Chen, X., Zhang, N., Li, L., Yao, Y., Deng, S., Tan, C., Huang, F., Si, L., & Chen, H. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.121 · Cited by 156
MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering
Gokhale, T., Banerjee, P., Baral, C. R., & Yang, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.63 · Cited by 161
Zhang, Z., Zhao, T., Guo, Y., & Yin, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3449154 · Cited by 208
SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition
Hu, H., Zhao, W., Zhou, W., Wang, Y., & Li, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01090 · Cited by 157
Airbert: In-domain Pretraining for Vision-and-Language Navigation
Guhur, P.-L., Tapaswi, M., Chen, S., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00166 · Cited by 139
TRAR: Routing the Attention Spans in Transformer for Visual Question Answering
Zhou, Y., Ren, T., Zhu, C., Sun, X., Liu, J., Ding, X., Xu, M., & Ji, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00208 · Cited by 133
STAR: A Structure-aware Lightweight Transformer for Real-time Image Enhancement
Zhang, Z., Jiang, Y., Jiang, J., Wang, X., Luo, P., & Gu, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00407 · Cited by 131
KAT: A Knowledge Augmented Transformer for Vision-and-Language
Gui, L., Wang, B., Huang, Q., Hauptmann, A., Bisk, Y., & Gao, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.70 · Cited by 130
Valipour, M., Rezagholizadeh, M., Kobyzev, I., & Ghodsi, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.239 · Cited by 304
What Does BERT with Vision Look At?
Li, L. H., Yatskar, M., Yin, D., Hsieh, C., & Chang, K.-W. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.469 · Cited by 141
Jin, W., Cheng, Y., Shen, Y., Chen, W., & Ren, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.197 · Cited by 174
Unifying Vision, Text, and Layout for Universal Document Processing
Tang, Z., Yang, Z., Wang, G., Fang, Y., Liu, Y., Zhu, C., Zeng, M., Zhang, C., & Bansal, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01845 · Cited by 164
Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
Zhuge, M., Gao, D., Fan, D.-P., Jin, L., Chen, B., Zhou, H., Qiu, M., & Shao, L. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01246 · Cited by 130
Towards Long-Form Video Understanding
Wu, C.-Y., & Krähenbühl, P. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00192 · Cited by 131
Using transformers for multimodal emotion recognition: Taxonomies and state of the art review
Hazmoune, S., & Bougamouza, F. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108339 · Cited by 136
Detecting Harmful Memes and Their Targets
Pramanick, S., Dimitrov, D., Mukherjee, R., Sharma, S., Akhtar, M. S., Nakov, P., & Chakraborty, T. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.246 · Cited by 132
Contrastive Learning for Weakly Supervised Phrase Grounding
Gupta, T., Vahdat, A., Chechik, G., Yang, X., Kautz, J., & Hoiem, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_44 · Cited by 125
Learning Visual Representations with Caption Annotations
Sarıyıldız, M. B., Pérez, J., & Larlus, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58598-3_10 · Cited by 128
Topological Planning with Transformers for Vision-and-Language Navigation
Chen, K., Chen, J. K., Chuang, J., Vázquez, M., & Savarese, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01112 · Cited by 130
CGMVQA: A New Classification and Generative Model for Medical Visual Question Answering
Ren, F., & Zhou, Y. (2020) · IEEE Access · doi:10.1109/access.2020.2980024 · Cited by 117
Federated Learning for Vision-and-Language Grounding Problems
Liu, F., Wu, X., Ge, S., Fan, W., & Zou, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6824 · Cited by 115
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
Yang, Z., Lu, Y., Wang, J., Yin, X. P., Florencio, D. A. F., Wang, L., Zhang, C., Zhang, L., & Luo, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00864 · Cited by 130
What Artificial Neural Networks Can Tell Us about Human Language Acquisition
Warstadt, A., & Bowman, S. R. (2022) · Algebraic Structures in Natural Language · doi:10.1201/9781003205388-2 · Cited by 111
HairCLIP: Design Your Hair by Text and Reference Image
Wei, T., Chen, D., Zhou, W., Liao, J., Tan, Z., Yuan, L., Zhang, W., & Yu, N. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01754 · Cited by 123
Self-Supervised Multimodal Learning: A Survey
Zong, Y., Aodha, O. M., & Hospedales, T. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429301 · Cited by 165
CLVIN: Complete language-vision interaction network for visual question answering
Chen, C., Han, D., & Shen, X. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110706 · Cited by 115
Peng, L., Jian, S., Kan, Z., Qiao, L., & Li, D. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103564 · Cited by 119
Efficient Self-supervised Vision Transformers for Representation LearningPreprint
Li, C., Yang, J., Zhang, P., Gao, M., Xiao, B., Dai, X., Yuan, L., & Gao, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09785 · Cited by 164
Referring Expression Comprehension: A Survey of Methods and Datasets
Qiao, Y., Deng, C., & Wu, Q. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3042066 · Cited by 115
Local-Global Context Aware Transformer for Language-Guided Video Segmentation
Liang, C., Wang, W., Zhou, T., Miao, J., Luo, Y., & Yang, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3262578 · Cited by 114
Bugliarello, E., Cotterell, R., Okazaki, N., & Elliott, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00408 · Cited by 133
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
Cao, J., Gan, Z., Cheng, Y., Yu, L., Chen, Y.-C., & Liu, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58539-6_34 · Cited by 112
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models
Li, X., Wang, C., Tang, Y., Tran, C., Tang, Y., Pino, J., Baevski, A., Conneau, A., & Auli, M. (2021) · doi:10.18653/v1/2021.acl-long.68 · Cited by 112
FashionVLP: Vision Language Transformer for Fashion Retrieval with Feedback
Goenka, S., Zheng, Z., Jaiswal, A., Chada, R., Wu, Y., Hedau, V., & Natarajan, P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01371 · Cited by 109
MixGen: A New Multi-Modal Data Augmentation
Hao, X., Zhu, Y., Appalaraju, S., Zhang, A., Zhang, W., Li, B., & Li, M. (2023) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw58289.2023.00042 · Cited by 110
Self-citationVideo Graph Transformer for Video Question Answering
Xiao, J., Zhou, P., Chua, T., & Yan, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_3 · Cited by 108
Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning
Kuo, C.-W., & Kira, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01744 · Cited by 99
Cross-media Structured Common Space for Multimedia Event Extraction
Li, M., Zareian, A., Zeng, Q., Whitehead, S., Lu, D., Ji, H., & Chang, S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.230 · Cited by 103
Human-Centric Spatio-Temporal Video Grounding With Visual Transformers
Tang, Z., Liao, Y., Liu, S., Li, G., Jin, X., Jiang, H., Yu, Q., & Xu, D. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3085907 · Cited by 116
TransVG++: End-to-End Visual Grounding With Language Conditioned Vision Transformer
Deng, J., Yang, Z., Liu, D., Chen, T., Zhou, W., Zhang, Y., Li, H., & Ouyang, W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3296823 · Cited by 106
Ishmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 107
LaTr: Layout-Aware Transformer for Scene-Text VQA
Biten, A. F., Litman, R., Xie, Y., Appalaraju, S., & Manmatha, R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01605 · Cited by 97
Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation
Yan, B., & Pei, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20204 · Cited by 94
TubeDETR: Spatio-Temporal Video Grounding with Transformers
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01595 · Cited by 114
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
Xu, H., Ghosh, G., Huang, P.-Y., Arora, P., Aminzadeh, M., Feichtenhofer, C., Metze, F., & Zettlemoyer, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.370 · Cited by 94
Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checking
Xu, H.-D., Li, Z., Zhou, Q. S., Li, C., Wang, Z., Cao, Y., Huang, H., & Mao, X.-L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.64 · Cited by 97
Self-citationDisentangling Hate in Online Memes
Lee, R. K.-W., Cao, R., Fan, Z., Jiang, J., & Chong, W.-H. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475625 · Cited by 98
VisualCOMET: Reasoning About the Dynamic Context of a Still Image
Park, J. S., Bhagavatula, C., Mottaghi, R., Farhadi, A., & Choi, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58558-7_30 · Cited by 102
XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding
Xu, Y., Lv, T., Cui, L., Wang, G., Lu, Y., Florencio, D. A. F., Zhang, C., & Wei, F. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.253 · Cited by 102
Self-citationFedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning
Chen, H., Zhang, Y., Krompaß, D., Gu, J., & Volker Tresp, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.29007 · Cited by 109
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
Sun, S., Chen, Y.-C., Li, L., Wang, S., Fang, Y., & Liu, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.77 · Cited by 94
Multi-stage Aggregated Transformer Network for Temporal Language Localization in Videos
Zhang, M., Yang, Y., Chen, X., Ji, Y., Xu, X., Li, J., & Shen, H. T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01248 · Cited by 91
Class-Agnostic Object Detection with Multi-modal Transformer
Maaz, M., Rasheed, H., Khan, S. H., Khan, F. S., Anwer, R. M., & Yang, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20080-9_30 · Cited by 87
ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval
Cheng, M., Sun, Y., Wang, L., Zhu, X., Yao, K., Chen, J., Song, G., Han, J., Liu, J., Ding, E., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00512 · Cited by 87
Self-citationVD-BERT: A Unified Vision and Dialog Transformer with BERT
Wang, Y., Joty, S., Lyu, M. R.-T., King, I., Xiong, C., & Hoi, S. C. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.269 · Cited by 103
Unified Pre-training with Pseudo Texts for Text-To-Image Person Re-identification
Shao, Z., Zhang, X., Ding, C., Wang, J., & Wang, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01026 · Cited by 95
Visualizing Transformers for NLP: A Brief Survey
Brașoveanu, A., & Andonie, R. (2020) · International Conference on Information Visualisation (IV) · doi:10.1109/iv51561.2020.00051 · Cited by 93
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
Xu, H., Yan, M., Li, C., Bi, B., Huang, S., Xiao, W., & Huang, F. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.42 · Cited by 84
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
Cho, J., Lu, J., Schwenk, D., Hajishirzi, H., & Kembhavi, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.707 · Cited by 99
Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries
Edwards, C., Zhai, C., & Ji, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.47 · Cited by 87
Zhu, X., Zhu, J., Li, H., Wu, X., Li, H., Wang, X., & Dai, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01630 · Cited by 108
Self-citationUplift and Upsample: Efficient 3D Human Pose Estimation with Uplifting Transformers
Einfalt, M., Ludwig, K., & Lienhart, R. W. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00292 · Cited by 87
Multi-Modal Machine Learning in Engineering Design: A Review and Future Directions
Song, B., Zhou, R., & Ahmed, F. (2023) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4063954 · Cited by 100
Liu, G., Liao, Y., Wang, F., Zhang, B., Zhang, L., Liang, X., Wan, X., Li, S., Li, Z., Zhang, S., et al. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3099165 · Cited by 81
Cross-modal Map Learning for Vision and Language Navigation
Georgakis, G., Schmeckpeper, K., Wanchoo, K., Dan, S., Miltsakaki, E., Roth, D., & Daniilidis, K. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01502 · Cited by 92
Dense Interaction Learning for Video-based Person Re-identification
He, T., Jin, X., Shen, X., Huang, J., Chen, Z., & Hua, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00152 · Cited by 81
Pano-AVQA: Grounded Audio-Visual Question Answering on 360° Videos
Yun, H., Yu, Y., Yang, W., Lee, K., & Kim, G. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00204 · Cited by 87
ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts
Cai, M., Liu, H., Mustikovela, S. K., Meyer, G. P., Chai, Y., Park, D., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01227 · Cited by 88
Attention mechanisms and deep learning for machine vision: A survey of the state of the artPreprint
Hafiz, A. M., Parah, S. A., & Bhat, R. U. A. (2021) · Research Square · doi:10.21203/rs.3.rs-510910/v1 · Cited by 92
Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
Tan, H. X., & Bansal, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.162 · Cited by 103
Qi, Y., Pan, Z., Hong, Y., Yang, M.-H., van den Hengel, A., & Wu, Q. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00168 · Cited by 80
CAAN: Context-Aware attention network for visual question answering
Chen, C., Han, D., & Chang, C. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108980 · Cited by 75
WebQA: Multihop and Multimodal QA
Chang, Y., Cao, G., Narang, M., Gao, J., Suzuki, H., & Bisk, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01600 · Cited by 110
Learning Joint Visual Semantic Matching Embeddings for Language-Guided Retrieval
Chen, Y., & Bazzani, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58542-6_9 · Cited by 74
Bilinear Graph Networks for Visual Question Answering
Guo, D., Xu, C., & Tao, D. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3104937 · Cited by 81
LFT-Net: Local Feature Transformer Network for Point Clouds Analysis
Gao, Y., Liu, X., Li, J., Fang, Z., Jiang, X., & Huq, K. M. S. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3140355 · Cited by 81
PromptCap: Prompt-Guided Image Captioning for VQA with GPT-3
Hu, Y., Hua, H., Yang, Z., Shi, W., Smith, N. A., & Luo, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00277 · Cited by 194
Learning to Generate Scene Graph from Natural Language Supervision
Zhong, Y., Shi, J., Yang, J., Xu, C., & Li, Y. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00184 · Cited by 73
Universal Spam Detection using Transfer Learning of BERT Model
Tida, V. S., & Hsu, S. (2022) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2022.921 · Cited by 96
Detecting and Understanding Harmful Memes: A Survey
Sharma, S. K., Alam, F., Akhtar, M. S., Dimitrov, D., Martino, G. D. S., Firooz, H., Halevy, A. Y., Silvestri, F., Nakov, P., & Chakraborty, T. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/781 · Cited by 109
Compressing Visual-linguistic Model via Knowledge Distillation
Fang, Z., Wang, J., Hu, X., Wang, L., Yang, Y., & Liu, Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00146 · Cited by 81
BridgeTower: Building Bridges between Encoders in Vision-Language Representation Learning
Xiao, X., Wu, C., Rosenman, S., Lal, V., Che, W., & Duan, N. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26263 · Cited by 91
LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling
Li, L., Gan, Z., Lin, K., Lin, C.-C., Liu, Z., Liu, C., & Wang, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02214 · Cited by 90
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning
Masry, A., Kavehzadeh, P., Long, X., Hoque, E., & Joty, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.906 · Cited by 180
Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization
Yu, T., Dai, W., Liu, Z., & Fung, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.326 · Cited by 78
Object-aware Video-language Pre-training for Retrieval
Wang, A. J., Ge, Y., Cai, G., Yan, R., Lin, X., Shan, Y., Qie, X., & Shou, M. Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00331 · Cited by 69
Transformer: A General Framework from Machine Translation to Others
Zhao, Y., Zhang, J., & Zong, C. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1393-5 · Cited by 77
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Dai, W., Hou, L., Shang, L., Jiang, X., Liu, Q., & Fung, P. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.187 · Cited by 107
Targeted Multimodal Sentiment Classification based on Coarse-to-Fine Grained Image-Target Matching
Yu, J., Wang, J., Xia, R., & Li, J. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/622 · Cited by 67
ViGT: proposal-free video grounding with a learnable token in the transformer
Li, K., Guo, D., & Wang, M. (2023) · Science China Information Sciences · doi:10.1007/s11432-022-3783-3 · Cited by 67
R-VQA: A robust visual question answering model
Chowdhury, S., & Soni, B. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112827 · Cited by 69
VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning
Hu, X., Yin, X. P., Lin, K., Zhang, L., Gao, J., Wang, L., & Liu, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16249 · Cited by 77
Metadata
- Authors
- Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J.
- Affiliations
- University of Science and Technology of China (CN)
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1908.08530
- OpenAlex
- W2969876226
- Cited by
- 950
- References
- 45
- Citations by year
- 2026: 2 · 2025: 47 · 2024: 170 · 2023: 255 · 2022: 194 · 2021: 197 · 2020: 80 · 2019: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- pre-training, multimodal transformer, region of interest features, visual commonsense reasoning, visual question answering, referring expression comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
