The DOI matches its DataCite record; 46 of the 46 references it cites were found in a registry or an index.
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Open accessAbstract
We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark. Code is released at \url{https://github.com/jackroos/VL-BERT}.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Vision Transformers are commonly evaluated using the sliding window procedure from MMSegmentation [59] (e.g., [10, 101, 103]).”
From MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022) · cites this paper as [59]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Flagged first, then by citations
Research works printed but not linked: 4
- Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , pp. 1440-1448, 2015.
- Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems , pp. 3294-3302, 2015.
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pp. 1097-1105, 2012.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 1(8), 2019.
Cited by
Sorted by citations
Items 201–300 of 957
Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks
Luo, G., Zhou, Y., Sun, X., Wang, Y., Cao, L., Wu, Y., Huang, F., & Ji, R. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3139234 · Cited by 67
Vision-Language Transformer for Interpretable Pathology Visual Question Answering
Naseem, U., Khushi, M., & Kim, J. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3163751 · Cited by 68
Scene Graph Refinement Network for Visual Question Answering
Qian, T., Chen, J., Chen, S., Wu, B., & Jiang, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3169065 · Cited by 69
Liu, J., Capurro, D., Nguyen, A., & Verspoor, K. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104466 · Cited by 66
Generative Bias for Robust Visual Question Answering
Cho, J. W., Kim, D.-J., Ryu, H., & Kweon, I. S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01124 · Cited by 69
Multitask Vision-Language Prompt Tuning
Shen, S., Yang, S., Zhang, T., Zhai, B., Gonzalez, J. E., Keutzer, K., & Darrell, T. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00556 · Cited by 85
FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
Rotstein, N., Bensaïd, D., Brody, S., Ganz, R., & Kimmel, R. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00559 · Cited by 66
Meta Module Network for Compositional Visual Reasoning
Chen, W., Gan, Z., Li, L., Cheng, Y., Wang, W. Y., & Liu, J. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00070 · Cited by 69
Robustness Verification for TransformersPreprint
Shi, Z., Zhang, H., Chang, K.-W., Huang, M., & Hsieh, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.06622 · Cited by 70
Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
Kuang, J., Shen, Y., Xie, J., Luo, H., Xu, Z., Li, R., Li, Y., Cheng, X., Lin, X., & Han, Y. (2025) · ACM Computing Surveys · doi:10.1145/3711680 · Cited by 84
Navgpt-2: Unleashing navigational reasoning capability for large vision-language models
Zhou, G., Hong, Y., Wang, Z., Wang, X. E., & Wu, Q. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72667-5_15 · Cited by 63
Violin: A Large-Scale Dataset for Video-and-Language Inference
Liu, J., Chen, W., Cheng, Y., Gan, Z., Yu, L., Yang, Y., & Liu, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01091 · Cited by 69
Medical Visual Question Answering via Conditional Reasoning and Contrastive Learning
Liu, B., Zhan, L.-M., Xu, L., & Wu, X.-M. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3232411 · Cited by 63
Kumar, G. K., & Nandakumar, K. (2022) · Workshop on NLP for Positive Impact (NLP4PI) · doi:10.18653/v1/2022.nlp4pi-1.20 · Cited by 111
Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding
Shi, F., Gao, R., Huang, W., & Wang, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3328185 · Cited by 73
Han, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 81
VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering
Wang, Y., Yasunaga, M., Ren, H., Wada, S., & Leskovec, J. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01973 · Cited by 68
Causal Reasoning Meets Visual Representation Learning: A Prospective Study
Liu, Y., Wei, Y.-S., Yan, H., Li, G., & Lin, L. (2022) · Machine Intelligence Research · doi:10.1007/s11633-022-1362-z · Cited by 68
BEST: BERT Pre-training for Sign Language Recognition with Coupling Tokenization
Zhao, W., Hu, H., Zhou, W., Shi, J., & Li, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25470 · Cited by 70
LAVIS: A One-stop Library for Language-Vision Intelligence
Li, D., Li, J., Le, H., Wang, G., Savarese, S., & Hoi, S. C. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.3 · Cited by 68
Vision-Language Models for Edge Networks: A Comprehensive Survey
Sharshar, A., Khan, L. U., Ullah, W., & Guizani, M. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3579032 · Cited by 92
RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER
Sun, L. J., Wang, J., Su, Y., Weng, F., Sun, Y., Zheng, Z., & Chen, Y. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.168 · Cited by 62
STVGBert: A Visual-linguistic Transformer based Framework for Spatio-temporal Video Grounding
Su, R., Yu, Q., & Xu, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00156 · Cited by 66
VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
Pârcălăbescu, L., Cafagna, M., Muradjan, L., Frank, A., Calixto, I., & Albert Gatt, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.567 · Cited by 112
Vision-Enhanced and Consensus-Aware Transformer for Image Captioning
Cao, S., An, G., Zheng, Z., & Wang, Z. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3178844 · Cited by 63
TSVFN: Two-Stage Visual Fusion Network for multimodal relation extraction
Zhao, Q., Gao, T., & Guo, N. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103264 · Cited by 63
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training
Zhou, M., Zhou, L., Wang, S., Cheng, Y., Li, L., Zhou, Y., & Liu, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00414 · Cited by 74
Contrastive Video Question Answering via Video Graph Transformer
Xiao, J., Zhou, P., Yao, A., Li, Y., Hong, R., Yan, S., & Chua, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292266 · Cited by 73
Kunlamai, T., Yamane, T., Suganuma, M., Chun, P., & Okatani, T. (2023) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.13086 · Cited by 61
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54
Image Difference Captioning With Instance-Level Fine-Grained Feature Representation
Huang, Q., Liang, Y., Wei, J., Cai, Y., Liang, H., Leung, H.-F., & Li, Q. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3074803 · Cited by 57
VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition
Tian, C., Wang, W., Zhu, X., Dai, J., & Qiao, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19806-9_5 · Cited by 65
Self-citationRobust Visual Question Answering: Datasets, Methods, and Future Challenges
Ma, J., Wang, P., Kong, D., Wang, Z., Liu, J., Pei, H., & Zhao, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3366154 · Cited by 64
Multimodal Joint Attribute Prediction and Value Extraction for E-commerce Product
Zhu, T., Wang, Y., Li, H., Wu, Y., He, X., & Zhou, B. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.166 · Cited by 74
Neural Collaborative Graph Machines for Table Structure Recognition
Liu, H., Li, X., Liu, B., Jiang, D., Liu, Y., & Ren, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00449 · Cited by 58
Context-aware Alignment and Mutual Masking for 3D-Language Pre-training
Jin, Z., Hayat, M., Yang, Y., Guo, Y., & Lei, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01057 · Cited by 56
LATTE: LAnguage Trajectory TransformEr
Bucker, A., Figueredo, L., Haddadin, S., Kapoor, A., Ma, S., Vemprala, S. H., & Bonatti, R. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161068 · Cited by 70
FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks
Xiao, H., Zhu, X., Yu, L., Zhang, L., Song, Y.-Z., & Xiang, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00262 · Cited by 58
Self-citationActive Contrastive Learning of Audio-Visual Video RepresentationsPreprint
Ma, S., Zeng, Z., McDuff, D., & Song, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09805 · Cited by 67
Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-Modal Pretraining
Zhan, X., Wu, Y., Dong, X., Wei, Y., Lu, M., Zhang, Y., Xu, H., & Liang, X. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01157 · Cited by 54
Progressive Language-Customized Visual Feature Learning for One-Stage Visual Grounding
Liao, Y., Zhang, A., Chen, Z., Hui, T., & Liu, S. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3181516 · Cited by 54
Context Disentangling and Prototype Inheriting for Robust Visual Grounding
Tang, W., Li, L., Liu, X., Jin, L., Tang, J., & Li, Z. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3339628 · Cited by 54
Scaling Vision-Language Models with Sparse Mixture of Experts
Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., & He, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.758 · Cited by 113
WeakCLIP: Adapting CLIP for Weakly-Supervised Semantic Segmentation
Zhu, L., Wang, X., Feng, J., Cheng, T., Li, Y., Jiang, B., Zhang, D., & Han, J. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02224-2 · Cited by 54
SAN: Side Adapter Network for Open-Vocabulary Semantic Segmentation
Xu, M., Zhang, Z., Wei, F., Hu, H., & Bai, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3311618 · Cited by 54
Self-citationAdversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Li, L., Lei, J., Gan, Z., & Liu, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00205 · Cited by 64
CoVR: Learning Composed Video Retrieval from Web Video Captions
Ventura, L., Yang, A., Schmid, C., & Varol, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28334 · Cited by 62
Kadiyala, L., Mermer, O., Samuel, R. D. J., Sermet, Y., & Demir, İ. (2024) · Hydrology · doi:10.3390/hydrology11090148 · Cited by 62
INVIGORATE: Interactive Visual Grounding and Grasping in Clutter
Zhang, H., Lu, Y., Yu, C., Hsu, D., Lan, X., & Zheng, N. (2021) · Robotics: Science and Systems · doi:10.15607/rss.2021.xvii.020 · Cited by 54
Temporal Reasoning via Audio Question Answering
Fayek, H. M., & Johnson, J. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3010650 · Cited by 57
Multimodal Pretraining for Dense Video Captioning
Huang, G., Pang, B., Zhu, Z., Rivera, C. E., & Soricut, R. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.48 · Cited by 56
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval
Geigle, G., Pfeiffer, J., Reimers, N., Vulić, I., & Gurevych, I. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00473 · Cited by 60
Reference-Based Defect Detection Network
Zeng, Z., Liu, B., Fu, J., & Chao, H. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3096067 · Cited by 55
Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering
Chen, L., Zheng, Y., Niu, Y., Zhang, H., & Xiao, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3290012 · Cited by 57
Self-supervised Calorie-aware Heterogeneous Graph Networks for Food Recommendation
Song, Y., Yang, X., & Xu, C. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3524618 · Cited by 52
From screens to scenes: A survey of embodied AI in healthcare
Liu, Y., Cao, X., Chen, T., Jiang, Y., You, J., Wu, M., Wang, X., Feng, M., Jin, Y., & Chen, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103033 · Cited by 76
Hierarchical Feature Aggregation Based on Transformer for Image-Text Matching
Dong, X., Zhang, H., Zhu, L., Nie, L., & Liu, L. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3164230 · Cited by 51
ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts
Lin, B., Zhu, Y., Chen, Z., Liang, X., Liu, J., & Liang, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01496 · Cited by 59
Bucker, A., Figueredo, L., Haddadinl, S., Kapoor, A., Ma, S., & Bonatti, R. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981810 · Cited by 55
ArtVerse: A Paradigm for Parallel Human–Machine Collaborative Painting Creation in Metaverses
Guo, C., Dou, Y., Bai, T., Dai, X., Wang, C., & Wen, Y. (2023) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2022.3230406 · Cited by 53
Lei, Z., Haq, A. U., Zeb, A., Suzauddola, M., & Zhang, D. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115708 · Cited by 49
Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network
Li, Y., Pan, Y., Yao, T., Chen, J., & Mei, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17034 · Cited by 49
Liu, D., Wang, T., Liu, S., Wang, R., Yao, S., & Abdelzaher, T. (2021) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn52240.2021.9522151 · Cited by 52
VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge
Ravi, S., Chinchure, A., Sigal, L., Liao, R., & Shwartz, V. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00121 · Cited by 54
Multi-Modal Interaction Graph Convolutional Network for Temporal Language Localization in Videos
Zhang, Z., Han, X., Song, X., Yan, Y., & Nie, L. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3113791 · Cited by 52
Towards Open-Vocabulary Scene Graph Generation with Prompt-Based Finetuning
He, T., Gao, L., Song, J., & Li, Y.-F. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19815-1_4 · Cited by 56
MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model
Ji, Y., Wang, J., Gong, Y., Zhang, L., Zhu, Y., Wang, H., Zhang, J., Sakai, T., & Yang, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02228 · Cited by 59
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 146
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization
Tang, Z., Lei, J., & Bansal, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.193 · Cited by 48
Parallel Attention Network with Sequence Matching for Video Grounding
Zhang, H., Sun, A., Jing, W., Zhen, L., Zhou, J. T., & Goh, S. M. R. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.69 · Cited by 61
Retrieval-augmented Image Captioning
Ramos, R., Elliott, D., & Martins, B. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.266 · Cited by 61
Which *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 57
A Package for Learning on Tabular and Text Data with Transformers
Gu, K., & Budhkar, A. (2021) · doi:10.18653/v1/2021.maiworkshop-1.10 · Cited by 54
FashionViL: Fashion-Focused Vision-and-Language Representation Learning
Xiao, H. M., Yu, L., Zhu, X., Zhang, L., Song, Y.-Z., & Xiang, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_37 · Cited by 47
Self-citationVision-and-Language Pretrained Models: A Survey
Long, S., Cao, F., Han, S. C., & Yang, H. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/773 · Cited by 66
Conditional Feature Learning Based Transformer for Text-Based Person Search
Gao, C., Cai, G., Jiang, X., Zheng, F., Zhang, J., Gong, Y., Lin, F., Sun, X., & Bai, X. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3205216 · Cited by 49
Zhou, W., Persello, C., Li, M., & Stein, A. (2023) · Remote Sensing of Environment · doi:10.1016/j.rse.2023.113767 · Cited by 51
Scaling-up medical vision-and-language representation learning with federated learning
Lu, S., Liu, Z., Liu, T., & Zhou, W. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107037 · Cited by 49
VideoXum: Cross-Modal Visual and Textural Summarization of Videos
Lin, J., Hua, H., Chen, M., Li, Y., Hsiao, J.-H., Ho, C., & Luo, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3335875 · Cited by 56
Mask Grounding for Referring Image Segmentation
Chng, Y. X., Zheng, H. W., Han, Y., Qiu, X., & Huang, G. M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02509 · Cited by 62
MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering
Urooj, A., Mazaheri, A., Lobo, N. D. V., & Shah, M. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.417 · Cited by 49
All You May Need for VQA are Image Captions
Changpinyo, S., Kukliansy, D., Szpektor, I., Chen, X., Ding, N., & Soricut, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.142 · Cited by 67
VisPhone: Chinese named entity recognition model enhanced by visual and phonetic features
Zhang, B., Cai, J., Zhang, H., & Shang, J. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103314 · Cited by 44
Zhang, Y., Pan, Y., Yao, T., Huang, R., Mei, T., & Chen, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00285 · Cited by 48
Zhang, C., Zhang, K., Pham, T. X., Niu, A., Qiao, Z., Yoo, C. D., & Kweon, I. S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01404 · Cited by 49
Uncurated Image-Text Datasets: Shedding Light on Demographic Bias
García, N., Hirota, Y., Wu, Y., & Nakashima, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00672 · Cited by 50
ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces
He, Z., Sunkara, S., Zang, X., Xu, Y., Liu, L., Wichers, N., Schubiner, G., Lee, R., & Chen, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16741 · Cited by 54
Scene-Intuitive Agent for Remote Embodied Visual Grounding
Lin, X., Li, G., & Yu, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00696 · Cited by 53
Cross-lingual Visual Pre-training for Multimodal Machine Translation
Çağlayan, O., Kuyu, M., Amac, M. S., Madhyastha, P., Erdem, E., Erdem, A., & Specia, L. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.112 · Cited by 47
PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models
Yao, Y., Chen, Q., Zhang, A., Ji, W., Liu, Z., Chua, T., & Sun, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.763 · Cited by 53
Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers
Frank, S., Bugliarello, E., & Elliott, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.775 · Cited by 70
Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding
Cao, Q., Li, B., Liang, X., Wang, K., & Lin, L. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3045034 · Cited by 44
Self-citationTransformer-Based Language-Person Search With Multiple Region Slicing
Li, H., Xiao, J., Sun, M., Lim, E. G., & Zhao, Y. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3073718 · Cited by 41
Adaptive Spatio-Temporal Graph Enhanced Vision-Language Representation for Video QA
Jin, W., Zhao, Z., Cao, X., Zhu, J., He, X., & Zhuang, Y. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3076556 · Cited by 41
Perception Matters: Detecting Perception Failures of VQA Models Using Metamorphic Testing
Yuan, Y., Wang, S., Jiang, M., & Chen, T. Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01663 · Cited by 41
Memobert: Pre-Training Model with Prompt-Based Learning for Multimodal Emotion Recognition
Zhao, J., Li, R., Jin, Q., Wang, X., & Li, H. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746910 · Cited by 39
Learning to Scale Multilingual Representations for Vision-Language Tasks
Burns, A., Kim, D., Wijaya, D. T., Saenko, K., & Plummer, B. A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_12 · Cited by 34
IntentVizor: Towards Generic Query Guided Interactive Video Summarization
Wu, G., Lin, J., & Silva, C. T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01025 · Cited by 39
Da, C., Wang, P., & Yao, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19815-1_19 · Cited by 39
Chen, T., Zhang, Z., Cheng, Y., Awadallah, A. H., & Wang, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01171 · Cited by 42
Metadata
- Authors
- Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J.
- Affiliations
- University of Science and Technology of China (CN)
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1908.08530
- OpenAlex
- W2969876226
- Cited by
- 950
- References
- 45
- Citations by year
- 2026: 2 · 2025: 47 · 2024: 170 · 2023: 255 · 2022: 194 · 2021: 197 · 2020: 80 · 2019: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- pre-training, multimodal transformer, region of interest features, visual commonsense reasoning, visual question answering, referring expression comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
