The DOI matches its DataCite record; 46 of the 46 references it cites were found in a registry or an index.
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Open accessAbstract
We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark. Code is released at \url{https://github.com/jackroos/VL-BERT}.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Vision Transformers are commonly evaluated using the sliding window procedure from MMSegmentation [59] (e.g., [10, 101, 103]).”
From MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022) · cites this paper as [59]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Flagged first, then by citations
Research works printed but not linked: 4
- Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , pp. 1440-1448, 2015.
- Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems , pp. 3294-3302, 2015.
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pp. 1097-1105, 2012.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 1(8), 2019.
Cited by
Sorted by citations
Items 901–957 of 957
Industrial Part Image Retrieval Combining Convolution and Attention Features
Shao, Q., Qi, Y., Yang, L., & Wu, B. (2022) · International Conference on Computer Engineering and Artificial Intelligence (ICCEAI) · doi:10.1109/icceai55464.2022.00056
Semi-supervised learning for visual relation annotation
Tajrobehkar, M. (2022) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/161913
Vision and Structured-Language Pretraining for Cross-Modal Food RetrievalPreprint
Shukor, M., Thome, N., & Cord, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.04267 · Cited by 1
Text-dominated multimodal text classification with quadruplet attention
Li, Y. (2022) · International Symposium on Artificial Intelligence and Robotics (ISAIR) · doi:10.1117/12.2665438
Is Cross-Modal Information Retrieval Possible Without Training?
Choi, H., Lee, H., Joe, S., & Gwon, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_27
Cross-modal representation learning and generation
Liu, H., Chen, J., Li, L., Bao, B., Li, Z., Liu, J., & Nie, L. (2023) · Journal of Image and Graphics · doi:10.11834/jig.230035 · Cited by 1
MVLP: Multi-perspective Vision-Language Pre-training Model for Ethically Aligned Meme DetectionPreprint
Zhang, N., Feng, X., Gu, T., & Chang, L. (2023) · doi:10.36227/techrxiv.23689437.v1 · Cited by 1
Scene understanding pipeline for maintenance oriented tasks automation
Zegaoui, Y., Dufour, S., & Bortolaso, C. (2023) · International Conference on Quality Control by Artificial Vision (QCAV) · doi:10.1117/12.2692213
Han, B., Jiang, X., Fang, Z., Fujita, H., & Gao, Y. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4526365
Kamigaito, H., Hayashi, K., & Watanabe, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.162 · Cited by 2
One does not fit all! On the Complementarity of Vision Encoders for Vision and Language Tasks
Geigle, G., Liu, C., Pfeiffer, J., & Gurevych, I. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.9
Recombination Samples Training for Robust Natural Language Visual Reasoning
Jiang, Y., Zhao, Y., & Bao, B. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00103
TraVLR: Now You See It, Now You Don’t! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoning
Chow, K. J., Tan, S., & Kan, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.242 · Cited by 1
Multimodal Pre-training Method for Vision-language Understanding and Generation
Liu, T., Wu, Z., Chen, J., & Jiang, Y. (2023) · International Journal of Software and Informatics · doi:10.21655/ijsi.1673-7288.00315
VGGAN: Visual Grounding GAN Using Panoptic Transformers
Quan, F., & Lang, B. (2023) · International Conference on Image, Vision and Computing (ICIVC) · doi:10.1109/icivc58118.2023.10270121
A survey of cross-modal pre-training
Sun, Y. (2023) · Applied and Computational Engineering · doi:10.54254/2755-2721/22/20231223
Decoupling-Cooperative Framework for Referring Expression Comprehension
Song, Y.-Z., Chen, Y.-S., & Shuai, H.-H. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3327651
A Contextualized Transformer-Based Method for Cyberbullying Detection
Rezvani, N., Beheshti, A., & Zhang, X. (2023) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa60987.2023.10302478
Kays, H. M. I., Momin, K. A., Muraleetharan, K. K., & Sadri, A. M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4622947
Literature Review: Recent Advances in Computer Vision and Language AI
Rajasekaran, S. B. (2023) · Journal of Artificial Intelligence & Cloud Computing · doi:10.47363/jaicc/2023(2)131
Text-Guided Image Restoration and Semantic Enhancement for Text-to-Image Person RetrievalPreprint
Liu, D., Li, H., Zhao, Z., Dong, Y., & Boulgouris, N. V. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4648005
Aesthetic Multi-attributes Captioning Network for Photos
Yang, H., Li, Y., Zhou, X., Jin, X., Shi, P. A., & Liu, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-99-9109-9_12
Leveraging Task-Specific Pre-Training to Reason across Images and Videos
Sadhu, A., & Nevatia, R. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00569
Lightening the Load: Lightweighting Multimodal Understanding for Visual Grounding Tasks
Pragati, P., & Singh, N. (2024) · International Conference on Data Engineering and Communication Systems (ICDECS) · doi:10.1109/icdecs59733.2023.10503418
Visual Questioner using Bert Transformer
Dr.Visu, P., Surya Narayanan, K., Syed Nafeez Ahmed, A., & Aravind Kumar, S. (2024) · International Journal of Science and Management Studies (IJSMS) · doi:10.51386/25815946/ijsms-v7i1p113
Semantic Completion: Enhancing Image-Text Retrieval with Information Extraction and Compression
Chen, X., & Guo, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2238-9_5
McInerney, D. J. (2024) · Digital Repository Service (Northeastern University) · doi:10.17760/d20653001
Distribution-Aware Prompt Generator for Long-Tailed Vision RecognitionPreprint
Liao, W., Fang, P., Ge, Z., & Harandi, M. T. (2024) · Research Square · doi:10.21203/rs.3.rs-4445913/v1
Cross-modal image text retrieval based on cross-optimization and joint strategy
Han, N., Zheng, W., Hu, Y., Kang, P., Zhang, Y., Yang, A., & Zhang, L. (2024) · International Conference on Image, Signal Processing and Pattern Recognition (ISPP) · doi:10.1117/12.3034096 · Cited by 1
Self-correcting text-aware strategy for text-caption
Lian, J., Cao, X., & Dai, P. (2024) · International Conference on Remote Sensing, Mapping, and Image Processing (RSMIP) · doi:10.1117/12.3029640
Bibliometric Innovation: Towards Work Training Literature In The Era Of Digital Development
Sucipto, I., & Kusumah, D. H. (2024) · Journal of Economic Bussines and Accounting (COSTING) · doi:10.31539/costing.v7i5.11675
HAPFI: History-Aware Planning based on Fused Information
Jeon, S., Shin, S., & Zhang, B. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610571
Structural and Contrastive Guidance Mining for Weakly-Supervised Language Moment Localization
Tang, D., & Cao, X. (2024) · IEEE Access · doi:10.1109/access.2024.3450878
A Multimodal Contrastive Network with Unbiased Distillation for Knowledge-based VQA
Hu, Z., Ding, R., Xie, H., & Yang, Z. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650444 · Cited by 1
Some Can Be Better than All: Multimodal Star Transformer for Visual Dialog
He, Q., Zhang, J., Qian, S., & Wang, C. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10647294
Research on the construction of language model based on improved neural network
Li, Z., Chi, S., Yuan, B., Bai, H., & Geng, X. (2024) · International Conference on Artificial Intelligence and Electromechanical Automation (AIEA) · doi:10.1109/aiea62095.2024.10692397
A Benchmark Dataset for Evaluating Spatial Perception in Multimodal Large Models
Xuan, L., Haoxiang, 张. Z., Baozheng, J., Yan-xia, L., & You, L. (2024) · doi:10.1145/3698385.3699875
TEIMA - Telecom Equipment Installation Multimodal Assistant
Sastry, N., Singh, B., Roychowdhury, S., Sharma, A., Gupta, A., & Mazumdar, S. (2025) · International Conference on Communication Systems & Networks (COMSNETS) · doi:10.1109/comsnets63942.2025.10885579
Boosting Movie and TV Tag Accuracy with Knowledge Graphs*
Jiang, H., Zhang, L., & Zhang, L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887686
Ma, M., Yu, Z., Ma, Y., Li, G., & Yang, Z. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887841
Curriculum-Guided Self-Supervised Representation Learning of Dynamic Heterogeneous Networks
Jung, N., Camacho, D., Choi, C., & Lee, O. (2025) · Cognitive Computation · doi:10.1007/s12559-025-10441-1
Iterative Multi-Label Semantic Cross-Fusion Network for Cross-Modal Hash RetrievalPreprint
Yuan, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5236421
VIKCSE: Visual-knowledge enhanced contrastive learning with prompts for sentence embedding
Cao, R., Wang, Y., Yan, H., Ren, J., Zheng, J., Jian-fen, Y., & Gao, L. C. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113659
Visual Question Answering Based on Object Detection and Recognition
Ashfaq, A., Ehatisham-ul-Haq, M., & Murtaza, F. (2024) · International Multi-Topic Conference · doi:10.1109/inmic64792.2024.11004332
Scene Graph Generation Approach with Integrated Environmental Contextual Information
Sun, Y., Zhu, Y., Tao, Y., & Li, B. (2025) · Asia Conference on Algorithms, Computing and Machine Learning (CACML) · doi:10.1109/cacml64929.2025.11010975
Self-citationEnhancing Visual Aligning and Grounding for Aerial Vision-and-Dialog Navigation
Qiao, G., Yi, D., Wu, L., Wu, H., & Wang, J. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3585831
A Multi-Modal Attentive Framework That Can Interpret Text (MMAT)
Kumari, V., Gupta, S., Sharma, Y., & Goel, L. (2025) · IEEE Access · doi:10.1109/access.2025.3586900 · Cited by 1
A Diffusion Scale-Enhanced CLIP Model for Cross-Lingual Cross-Modal Building Information Retrieval
Tang, M., Zhang, L., & Yang, C. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9921-6_29
Explainability of Language Models (XAI and LLM)
Cecco, A. D., & Gianfagna, L. (2025) · Springer eBooks · doi:10.1007/978-3-031-92229-9_8
Zhan, H., Song, Y., Huang, X., Tan, X. J., & Zhang, T. (2025) · Frontiers in Remote Sensing · doi:10.3389/frsen.2025.1662024 · Cited by 1
Text-enhanced cross-modal reinforced fusion network for multimodal sentiment analysis
Zhao, W., & Wu, Y. (2026) · Pattern Analysis and Applications · doi:10.1007/s10044-026-01676-1
Xu, R., Yao, L., Wu, Z., Zhang, X., & Wu, G. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810607
Multimodal Few-Shot Learning with Frozen Language Models
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A., Vinyals, O., & Hill, F. (2021) · arXiv · doi:10.48550/arxiv.2106.13884 · Cited by 355
Chasing Sparsity in Vision Transformers: An End-to-End Exploration
Chen, T., Cheng, Y., Gan, Z., Yuan, L., Zhang, L., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2106.04533 · Cited by 111
Metadata
- Authors
- Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J.
- Affiliations
- University of Science and Technology of China (CN)
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1908.08530
- OpenAlex
- W2969876226
- Cited by
- 950
- References
- 45
- Citations by year
- 2026: 2 · 2025: 47 · 2024: 170 · 2023: 255 · 2022: 194 · 2021: 197 · 2020: 80 · 2019: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- pre-training, multimodal transformer, region of interest features, visual commonsense reasoning, visual question answering, referring expression comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
