The DOI matches its DataCite record; 46 of the 46 references it cites were found in a registry or an index.
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Open accessAbstract
We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark. Code is released at \url{https://github.com/jackroos/VL-BERT}.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Vision Transformers are commonly evaluated using the sliding window procedure from MMSegmentation [59] (e.g., [10, 101, 103]).”
From MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022) · cites this paper as [59]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Flagged first, then by citations
Research works printed but not linked: 4
- Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , pp. 1440-1448, 2015.
- Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems , pp. 3294-3302, 2015.
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pp. 1097-1105, 2012.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 1(8), 2019.
Cited by
Sorted by citations
Items 501–600 of 957
CAPT: Contrastive Pre-Training for Learning Denoised Sequence RepresentationsPreprint
Luo, F., Yang, P., Li, S., Ren, X., & Sun, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06351 · Cited by 17
Classification of Multimodal Hate Speech -- The Winning Solution of Hateful Memes ChallengePreprint
Zhong, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01002 · Cited by 18
Li, Z., Fan, Z., Chen, J., Zhang, Q., Huang, X., & Wei, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.327 · Cited by 17
Enhanced Chart Understanding via Visual Language Pre-training on Plot Table Pairs
Zhou, M., Fung, Y. R., Chen, L., Thomas, C., Ji, H., & Chang, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.85 · Cited by 15
SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining
Song, C. H., Hwang, T., Yoon, J., Choi, S., & Gu, Y. H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01323 · Cited by 14
Adapting Vision-Language Models via Learning to Inject Knowledge
Xuan, S., Yang, M., & Zhang, S. S. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3468884 · Cited by 15
Probing Multi-modal Machine Translation with Pre-trained Language Model
Kong, Y., & Fan, K. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.323 · Cited by 13
Talk2Car: Predicting Physical Trajectories for Natural Language Commands
Deruyttere, T., Grujicic, D., Blaschko, M. B., & Moens, M. (2022) · IEEE Access · doi:10.1109/access.2022.3224144 · Cited by 13
Relaxing Contrastiveness in Multimodal Representation Learning
Lin, Z., Bas, E., Singh, K. Y., Gurumurthy, S., & Bhotika, R. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00226 · Cited by 14
Multi-Modal Structure-Embedding Graph Transformer for Visual Commonsense Reasoning
Zhu, J., Wang, H., & He, B. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3279691 · Cited by 14
FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning
Mirchandani, S., Yu, L., Wang, M., Sinha, A., Jiang, W., Xiang, T., & Zhang, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.716 · Cited by 13
MMVAE at SemEval-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection
Gu, Y., Castro, I., & Gareth Tyson, A. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.96 · Cited by 12
Complete 3D Relationships Extraction Modality Alignment Network for 3D Dense Captioning
Mao, A., Yang, Z. X., Chen, W., Yi, R., & Liu, Y. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3279204 · Cited by 12
Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval
Dong, X. M., Zhan, X., Wei, Y., Wei, X.-Y., Wang, Y., Lu, M., Cao, X., & Liang, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3291237 · Cited by 14
Improving visual grounding with multi-scale discrepancy information and centralized-transformer
Wu, J., Wu, C., Wang, F., Wang, L., & Wei, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123223 · Cited by 12
Multi-Modal Hierarchical Empathetic Framework for Social Robots With Affective Body Control
Gao, Y., Fu, Y., Sun, M., & Gao, F. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3356511 · Cited by 12
Qin, J., Liu, C., Cheng, S. L., Guo, Y., & Arcucci, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447326 · Cited by 17
LVAR-CZSL: Learning Visual Attributes Representation for Compositional Zero-Shot Learning
Ma, X., Yang, J., Lin, J., Zheng, Z., Li, S., Hu, B., & Tang, X. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3444782 · Cited by 12
MAFA: Managing False Negatives for Vision-Language Pre-Training
Byun, J., Kim, D., & Moon, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02579 · Cited by 13
Multimodal large model pretraining, adaptation and efficiency optimization
Ji, L., Xiao, S., Feng, J., Gao, W., & Zhang, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129138 · Cited by 14
Text-guided Image Restoration and Semantic Enhancement for Text-to-Image Person Retrieval
Liu, D., Li, H., Zhao, Z., & Dong, Y. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107028 · Cited by 13
Context-Sensitive Visualization of Deep Learning Natural Language Processing Models
Dunn, A., Inkpen, D. Z., & Andonie, R. (2021) · International Conference on Information Visualisation (IV) · doi:10.1109/iv53921.2021.00035 · Cited by 12
Visual commonsense reasoning with directional visual connections
Han, Y., Wu, A., Zhu, L., & Yang, Y. (2021) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2000722 · Cited by 12
Hateful Memes Classification using Machine Learning
Badour, J., & Brown, J. A. (2021) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci50451.2021.9659896 · Cited by 12
Position-aware image captioning with spatial relation
Duan, Y., Wang, Z., Wang, J., Wang, Y., & Lin, C. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.05.003 · Cited by 13
Accurate Solar Wind Speed Prediction with Multimodality Information
Sun, Y., Xie, Z., Chen, Y., & Hu, Q. (2022) · Space Science & Technology · doi:10.34133/2022/9805707 · Cited by 12
Bilinear Semi-Tensor Product Attention (BSTPA) model for visual question answering
Bai, Z., Li, Y., Zhou, M., Li, D., Wang, D., Połap, D., & Woźniak, M. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206964 · Cited by 12
Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision
Pezzelle, S., Greco, C., Gandolfi, G., Gualdoni, E., & Bernardi, R. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.248 · Cited by 15
Multi-stage Pre-training over Simplified Multimodal Pre-training Models
Liu, T., Feng, F., & Wang, X. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.199 · Cited by 11
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
Miech, A., Alayrac, J.-B., Laptev, I., Sivic, J., & Zisserman, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00970 · Cited by 149
Who’s Waldo? Linking People Across Text and Images
Cui, C. Y., Khandelwal, A., Artzi, Y., Snavely, N., & Averbuch‐Elor, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00141 · Cited by 11
A Critical Analysis of Benchmarks, Techniques, and Models in Medical Visual Question Answering
Al-Hadhrami, S. A., Menaï, M. E. B., Al-Ahmadi, S. A., & Alnafessah, A. (2023) · IEEE Access · doi:10.1109/access.2023.3335216 · Cited by 14
Hierarchical reasoning based on perception action cycle for visual question answering
Mohamud, S. A. M., Jalali, A., & Lee, M. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122698 · Cited by 11
LOIS: Looking Out of Instance Semantics for Visual Question Answering
Zhang, S., Chen, Y., Sun, Y., Wang, F., Shi, H., & Wang, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3347093 · Cited by 12
A Retrospect to Multi-prompt Learning across Vision and Language
Chen, Z., Huang, X., Guan, Q., Lin, L., & Luo, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02028 · Cited by 11
Learning to Learn Better Visual Prompts
Wang, F., Huang, W., Yang, S., Fan, Q., & Lan, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28343 · Cited by 12
Cycle-Consistency Learning for Captioning and Grounding
Wang, N., Deng, J., & Jia, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28363 · Cited by 11
CoSTA: Co-training spatial–temporal attention for blind video quality assessment
Xing, F., Wang, Y., Tang, W., Zhu, G., & Kwong, S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124651 · Cited by 11
Liu, Y., Xiao, F., Zheng, X., Deng, W., Ma, H., Su, X., & Wu, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-85467-4 · Cited by 11
TIRA in Baidu Image Advertising
Yu, T., Yang, X., Jiang, Y., Zhang, H., Zhao, W., & Li, P. (2021) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde51399.2021.00225 · Cited by 11
Constructing Multi-Modal Dialogue Dataset by Replacing Text with Semantically Relevant Images
Lee, N., Shin, S., Choo, J., Choi, H., & Myaeng, S.-H. (2021) · doi:10.18653/v1/2021.acl-short.113 · Cited by 12
Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imagination
Yang, Y., Yao, W., Zhang, H., Wang, X., Yu, D., & Chen, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.78 · Cited by 17
Commands for Autonomous Vehicles by Progressively Stacking Visual-Linguistic Representations
Dai, H., Luo, S., Ding, Y. S., & Shao, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66096-3_2 · Cited by 11
MLNet: a multi-level multimodal named entity recognition architecture
Zhai, H., Lv, X., Hou, Z., Tong, X., & Bu, F. (2023) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2023.1181143 · Cited by 10
Online Query Expansion Hashing for Efficient Image Retrieval
Cui, H. J., Li, F., Zhu, L., Li, J., & Zhang, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3296412 · Cited by 12
Text-Guided Face Recognition using Multi-Granularity Cross-Modal Contrastive Learning
Hasan, M. M., Sami, S. M., & Nasrabadi, N. M. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00568 · Cited by 10
MAC: Masked Contrastive Pre-Training for Efficient Video-Text Retrieval
Shu, F., Chen, B., Liao, Y., Wang, J., & Liu, S. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3402613 · Cited by 23
UNK-VQA: A Dataset and a Probe Into the Abstention Ability of Multi-Modal Large Models
Guo, Y., Jiao, F., Shen, Z., Nie, L., & Kankanhalli, M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3437288 · Cited by 18
TagCLIP: Improving Discrimination Ability of Zero-Shot Semantic Segmentation
Li, J., Chen, P., Qian, S., Liu, S., & Jia, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3454647 · Cited by 21
Aesthetic image cropping meets VLP: Enhancing good while reducing bad
Yuan, Q., Li, L., & Chen, P. (2024) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2024.104316 · Cited by 10
Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection
Xu, Y., Zhang, M., Yang, X., & Xu, C. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3485518 · Cited by 17
LININ: Logic Integrated Neural Inference Network for Explanatory Visual Question Answering
Xue, D., Qian, S., Fang, Q., & Xu, C. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521709 · Cited by 12
Is CLIP the main roadblock for fine-grained open-world perception?
Bianchi, L., Carrara, F., Messina, N., & Falchi, F. (2024) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi62980.2024.10859215 · Cited by 11
BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis
Monajatipoor, M., Rouhsedaghat, M., Li, L. H., Chien, A., Kuo, C. ‐. J., Scalzo, F., & Chang, K.-W. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00372 · Cited by 13
ASRtrans at SemEval-2022 Task 5: Transformer-based Models for Meme Classification
Rao, A. R., & Rao, A. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.82 · Cited by 10
Guo, P., Song, Y., Deng, Y., Xie, K., Xu, M., Liu, J., & Ren, H. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3287296 · Cited by 10
Leveraging per Image-Token Consistency for Vision-Language Pre-training
Gou, Y., Ko, T., Yang, H., Kwok, J. T.-Y., Zhang, Y., & Wang, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01836 · Cited by 11
Cai, Y., Thomason, J., & Rostami, M. G. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.466 · Cited by 10
Human-centric Relation Segmentation: Dataset and Solution
Liu, S., Wang, Z., Gao, Y., Ren, L., Liao, Y., Ren, G., Li, B. H., & Yan, S. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3075846 · Cited by 12
Self-citationResuFormer: Semantic Structure Understanding for Resumes via Multi-Modal Pre-training
Yao, K., Zhang, J., Qin, C., Song, X. Y., Wang, P., Zhu, H., & Xiong, H. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00242 · Cited by 12
What BERT Sees: Cross-Modal Transfer for Visual Question Generation
Scialom, T., Bordes, P., Dray, P.-A., Staiano, J., & Gallinari, P. (2020) · Proceedings of the 13th International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.39 · Cited by 9
Discovering and Mitigating Biases in CLIP-based Image Editing
Tanjim, M. M., Singh, K. K., Kafle, K., Sinha, R., & Cottrell, G. W. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00296 · Cited by 11
Progressive Semantic Reconstruction Network for Weakly Supervised Referring Expression Grounding
Ji, Z., Wu, J., Wang, Y., Yang, A., & Han, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3433547 · Cited by 9
Wen, S., Gong, S., Zhang, Z., Yu, F. R., & Wang, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112610 · Cited by 10
A Survey of Human-Object Interaction Detection With Deep Learning
Geng, H., Zhao, J., Zhang, L., & Deng, F. Y. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3518613 · Cited by 11
In Factuality: Efficient Integration of Relevant Facts for Visual Question Answering
Vickers, P., Αλέτρας, Ν., Monti, E., & Barrault, L. (2021) · doi:10.18653/v1/2021.acl-short.60 · Cited by 9
PiGLET: Pixel-Level Grounding of Language Expressions With Transformers
González, C., Ayobi, N., Hernández, I., Pont-Tuset, J., & Arbeláez, P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3286760 · Cited by 9
Zhu, Y., Zhu, Z., Lin, B., Liang, X., Zhao, F., & Liu, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.317 · Cited by 9
Data-Efficient Multimodal Fusion on a Single GPU
Vouitsis, N., Liu, Z., Gorti, S. K., Villecroze, V., Cresswell, J. C., Yu, G., Loaiza-Ganem, G., & Volkovs, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02572 · Cited by 11
Cross-modal Relational Reasoning Network for Visual Question Answering
Chen, H., Liu, R., & Peng, B. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00441 · Cited by 9
Feature-guided Multimodal Sentiment Analysis towards Industry 4.0
Yu, B., Wei, J., Yu, B., Cai, X., Wang, K., Sun, H., Bu, L., & Chen, X. (2022) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2022.107961 · Cited by 10
Hirasawa, T., Kaneko, M., Imankulova, A., & Komachi, M. (2022) · IEEE Access · doi:10.1109/access.2022.3185243 · Cited by 9
Multimodal Inverse Cloze Task for Knowledge-Based Visual Question Answering
Lerner, P., Ferret, O., & Guinaudeau, C. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_36 · Cited by 12
DimCL: Dimensional Contrastive Learning for Improving Self-Supervised Learning
Nguyen, T., Pham, T. X., Zhang, C., Luu, T. M., Vu, T., & Yoo, C. D. (2023) · IEEE Access · doi:10.1109/access.2023.3236087 · Cited by 13
InterBERT: An Effective Multi-Modal Pretraining Approach via Vision-and-Language InteractionPreprint
Lin, J., An, Y., Zhang, Y., Liu, J., Zhou, J., & Yang, H. (2020) · arXiv (Cornell University) · Cited by 11
A Priority Map for Vision-and-Language Navigation with Trajectory Plans and Feature-Location Cues
Armitage, J., Impett, L., & Sennrich, R. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00115 · Cited by 8
Fine-Tuning Multimodal Transformer Models for Generating Actions in Virtual and Real Environments
Staroverov, A., Gorodetsky, A., Krishtopik, A. S., Izmesteva, U. A., Yudin, D. A., Kovalev, A. K., & Panov, A. I. (2023) · IEEE Access · doi:10.1109/access.2023.3334791 · Cited by 11
Liu, X., Dong, R., & Yang, H. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3453919 · Cited by 8
Graph-Based Group Division Network for Referring Expression Comprehension
Ke, J., Wang, J., Wong, W. K., Toomey, A., & Wen, J. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3528657 · Cited by 8
Finding Structural Knowledge in Multimodal-BERT
Milewski, V., de Lhoneux, M., & Moens, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.388 · Cited by 8
Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manuals
Wu, T.-L., Spangher, A., Alipoormolabashi, P., Freedman, M., Weischedel, R., & Peng, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.310 · Cited by 11
Towards Cross-Lingual Multi-Modal Misinformation Detection for E-Commerce Management
He, Y., Li, Z., Li, Z., Zhou, S., Yu, T., & Zhang, J. (2023) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2023.3234114 · Cited by 10
Cross-domain multi-style merge for image captioning
Duan, Y., Wang, Z., Li, Y., & Wang, J. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2022.103617 · Cited by 8
Learning to Imagine: Visually-Augmented Natural Language Generation
Tang, T., Chen, Y., Du, Y., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.526 · Cited by 12
Prompt Tuning for Unified Multimodal Pretrained Models
Yang, H., Lin, J., Yang, A., Wang, P., & Zhou, C. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.27 · Cited by 12
Kalarani, A. R., Bhattacharyya, P., Chhaya, N., & Shekhar, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.67 · Cited by 8
Vision-Text Cross-Modal Fusion for Accurate Video Captioning
Ouenniche, K., Țapu, R., & Zaharia, T. B. (2023) · IEEE Access · doi:10.1109/access.2023.3324052 · Cited by 8
ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation
Wang, W., Yang, Z., Xu, B., Li, J., & Sun, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00293 · Cited by 10
BUS : Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarization
Jiang, C., Xu, H., Ye, W., Ye, Q., Li, C., Yan, M., Bi, B., Zhang, S., Huang, F., & Huang, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00271 · Cited by 9
Multimodal Distillation Pre-Training Model for Ultrasound Dynamic Images Annotation
Chen, X., Jia, K., Zhang, Y., Gou, J., Shen, A., & Wan, S. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3438254 · Cited by 8
Learning Visual Grounding from Generative Vision and Language Model
Wang, S., Kim, D., Taalimi, A., Sun, C., & Kuo, W. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00782 · Cited by 18
Utilizing Every Image Object for Semi-supervised Phrase Grounding
Zhu, H., Sadhu, A., Zheng, Z., & Nevatia, R. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00226 · Cited by 8
AIMH at SemEval-2021 Task 6: Multimodal Classification Using an Ensemble of Transformer Models
Messina, N., Falchi, F., Gennaro, C., & Amato, G. M. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.140 · Cited by 9
Detecting Persuasive Atypicality by Modeling Contextual Compatibility
Guo, M., Hwa, R., & Kovashka, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00101 · Cited by 12
Latent Variable Models for Visual Question Answering
Wang, Z., Miao, Y., & Specia, L. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00351 · Cited by 8
Rethinking Denoised Auto-Encoding in Language Pre-Training
Luo, F., Yang, P., Li, S., Ren, X., Sun, X., Huang, S., & Huang, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.232 · Cited by 8
Detecting the Role of an Entity in Harmful Memes: Techniques and their Limitations
Nandi, R. N., Alam, F., & Nakov, P. (2022) · doi:10.18653/v1/2022.constraint-1.6 · Cited by 8
Dealing with Semantic Underspecification in Multimodal NLP
Pezzelle, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.675 · Cited by 14
A Light Touch Approach to Teaching Transformers Multi-view Geometry
Bhalgat, Y. S., Henriques, J. F., & Zisserman, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00480 · Cited by 8
Metadata
- Authors
- Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J.
- Affiliations
- University of Science and Technology of China (CN)
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1908.08530
- OpenAlex
- W2969876226
- Cited by
- 950
- References
- 45
- Citations by year
- 2026: 2 · 2025: 47 · 2024: 170 · 2023: 255 · 2022: 194 · 2021: 197 · 2020: 80 · 2019: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- pre-training, multimodal transformer, region of interest features, visual commonsense reasoning, visual question answering, referring expression comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
