The DOI matches its DataCite record; 46 of the 46 references it cites were found in a registry or an index.
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Open accessAbstract
We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark. Code is released at \url{https://github.com/jackroos/VL-BERT}.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Vision Transformers are commonly evaluated using the sliding window procedure from MMSegmentation [59] (e.g., [10, 101, 103]).”
From MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022) · cites this paper as [59]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Research works printed but not linked: 4
- Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , pp. 1440-1448, 2015.
- Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems , pp. 3294-3302, 2015.
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pp. 1097-1105, 2012.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 1(8), 2019.
Cited by
Sorted by citations
Items 301–400 of 957
SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning
Wang, Z., You, H., Li, L. H., Zareian, A., Park, S., Liang, Y., Chang, K.-W., & Chang, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20536 · Cited by 43
Playing Lottery Tickets with Vision and Language
Gan, Z., Chen, Y.-C., Li, L., Chen, T., Cheng, Y., Wang, S., Liu, J., Wang, L., & Liu, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19945 · Cited by 48
GilBERT: Generative Vision-Language Pre-Training for Image-Text Retrieval
Hong, W., Ji, K., Liu, J., Wang, J., Chen, J., & Chu, W. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462838 · Cited by 37
LGR-NET: Language Guided Reasoning Network for Referring Expression Comprehension
Lu, M., Li, R., Feng, F., Ma, Z., & Wang, X. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3374786 · Cited by 39
Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
Li, L. H., You, H., Wang, Z., Zareian, A., Chang, S., & Chang, K.-W. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.420 · Cited by 51
Positional Attention Guided Transformer-Like Architecture for Visual Question Answering
Mao, A., Yang, Z. X., Lin, K., Xuan, J., & Liu, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3216770 · Cited by 37
Pan, X., Li, X., Li, Q., Hu, Z., & Bao, J. (2024) · Journal of Engineering Design · doi:10.1080/09544828.2023.2301230 · Cited by 40
Word2Pix: Word to Pixel Cross-Attention Transformer in Visual Grounding
Zhao, H., Zhou, J. T., & Ong, Y.-S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3183827 · Cited by 38
Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal TransformersPreprint
Yang, R., Zhang, M., Hansen, N., Xu, H., & Wang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03996 · Cited by 76
Attention-Enhanced Multimodal Learning for Conceptual Design Evaluations
Song, B., Miller, S. R., & Ahmed, F. (2023) · Journal of Mechanical Design · doi:10.1115/1.4056669 · Cited by 46
OpenViVQA: Task, dataset, and multimodal fusion models for visual question answering in Vietnamese
Nguyen, N. H., Vo, D. T. D., Nguyen, K. V., & Nguyen, N. L.-T. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101868 · Cited by 41
SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representation
Li, Z., Kim, J., Chiang, Y., & Chen, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.200 · Cited by 59
EarthVQANet: Multi-task visual question answering for remote sensing image understanding
Wang, J., Ma, A., Chen, Z., Zheng, Z., Wan, Y., Zhang, L., & Zhong, Y. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.05.001 · Cited by 45
Separating Skills and Concepts for Novel Visual Question Answering
Whitehead, S., Wu, H., Ji, H., Feris, R., & Saenko, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00558 · Cited by 34
KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for visual commonsense reasoning
Song, D., Ma, S., Sun, Z., Yang, S., & Liao, L. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107408 · Cited by 42
Universal Multimodal Representation for Language Understanding
Zhang, Z., Chen, K., Wang, R., Utiyama, M., Sumita, E., Li, Z., & Zhao, H. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3234170 · Cited by 39
Adapt and explore: Multimodal mixup for representation learning
Lin, R., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102216 · Cited by 34
A survey of methods for addressing the challenges of referring image segmentation
Ji, L., Du, Y., Dang, Y., Gao, W., & Zhang, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127599 · Cited by 35
GT-Finder: Classify the family of glucose transporters with pre-trained BERT language models
Shah, S. M. A., Taju, S. W., Ho, Q., Nguyen, T., & Ou, Y. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104259 · Cited by 34
Feng, J., Wang, G., Zheng, C., Cai, Y., Fu, Z., Wang, Y., Wei, X.-Y., & Li, Q. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3284474 · Cited by 35
Guo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 58
A Case Study of the Shortcut Effects in Visual Commonsense Reasoning
Ye, K., & Kovashka, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16428 · Cited by 34
i-Code: An Integrative and Composable Multimodal Learning Framework
Yang, Z., Fang, Y., Zhu, C., Pryzant, R., Chen, D., Shi, Y., Yi‐chong, X., Qian, Y., Gao, M., Chen, Y., et al. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26290 · Cited by 45
Self-citationSymbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task
Lei, S. W., Gao, D., Wu, J. Z., Wang, Y., Liu, W., Zhang, M., & Shou, M. Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25208 · Cited by 39
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training
Dai, W., Liu, Z., Ji, Z., Su, D., & Fung, P. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.156 · Cited by 59
An Overview of Vision Transformers for Image Processing: A Survey
Kameswari, C. S., Kavitha, J., Reddy, T. S., Chinthaguntla, B., Jagatheesaperumal, S. K., Gaftandzhieva, S. N., & Doneva, R. Z. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140830 · Cited by 38
GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding
Li, Z., Zhou, W., Chiang, Y., & Chen, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.317 · Cited by 49
Cross-Modality Relevance for Reasoning on Language and Vision
Chen, Z., Guo, Q., & Kordjamshidi, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.683 · Cited by 31
Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models
Srinivasan, T., & Bisk, Y. (2022) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2022.gebnlp-1.10 · Cited by 53
SSAN: Separable Self-Attention Network for Video Representation Learning
Guo, X., Guo, X., & Lu, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01243 · Cited by 35
COM: Contrastive Masked-attention model for incomplete multimodal learning
Qian, S., & Wang, C. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.03.003 · Cited by 33
Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts
Chen, Z., Diao, S., Wang, B., Li, G., & Wan, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02139 · Cited by 34
Pre-Trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness
Wang, S., Zhang, J., Yuan, Z., & Shan, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02313 · Cited by 41
Visual Question Answering on 360° Images
Chou, S.-H., Chao, W., Lai, W., Sun, M., & Yang, M. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093452 · Cited by 33
Are Vision-Language Transformers Learning Multimodal Representations? A Probing Perspective
Salin, E., Farah, B., Ayache, S., & Favre, B. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21375 · Cited by 37
VisQA: X-raying Vision and Language Reasoning in Transformers
Jaunet, T., Kervadec, C., Vuillemot, R., Antipov, G., Baccouche, M., & Wolf, C. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114683 · Cited by 30
“This work focuses on detector-based VL transformers such as LXMERT, however other VQA systems may include single stream models such as [14, 53].”
MethodMapping medical image-text to a joint space via masked modeling
Chen, Z., Du, Y., Hu, J., Liu, Y., Li, G., Wan, X., & Chang, T. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103018 · Cited by 31
A Study of Pre-trained Language Models in Natural Language Processing
Duan, J., Zhao, H. J., Zhou, Q., Qiu, M., & Liu, M. (2020) · IEEE International Conference on Smart Cloud (SmartCloud) · doi:10.1109/smartcloud49737.2020.00030 · Cited by 32
Contrast and Classify: Training Robust VQA Models
Kant, Y., Moudgil, A., Batra, D., Parikh, D., & Agrawal, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00163 · Cited by 34
Ye, S., Chen, D., Han, S., & Liao, J. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3225327 · Cited by 38
Unsupervised Vision-and-Language Pretraining via Retrieval-based Multi-Granular Alignment
Zhou, M., Yu, L., Singh, A., Wang, M., Zhou, Y., & Zhang, N. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01599 · Cited by 32
Connecting What to Say With Where to Look by Modeling Human Attention Traces
Meng, Z., Yu, L., Zhang, N., Berg, T. L., Damavandi, B., Singh, V., & Bearman, A. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01249 · Cited by 27
Transitional Adaptation of Pretrained Models for Visual Storytelling
Yu, Y., Chung, J., Yun, H., Kim, J.-S., & Kim, G. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01247 · Cited by 30
Wang, T., Zhou, W., Zeng, Y., & Zhang, X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.873 · Cited by 67
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive LearningPreprint
Li, W., Gao, C., Niu, G., Xiao, X., Líu, H., Liu, J., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15409 · Cited by 29
Mumuni, F., & Mumuni, A. (2023) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101188 · Cited by 34
MCPL: Multi-Modal Collaborative Prompt Learning for Medical Vision-Language Model
Wang, P., Zhang, H., & Yuan, Y. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3418408 · Cited by 28
Zero-Shot Video Moment Retrieval With Angular Reconstructive Text Embeddings
Jiang, X., Xu, X., Zhou, Z., Yang, Y., Shen, F., & Shen, H. T. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3396272 · Cited by 28
A survey on knowledge-enhanced multimodal learning
Lymperaiou, M., & Stamou, G. B. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10825-z · Cited by 45
Cross-Modal Remote Sensing Image–Text Retrieval via Context and Uncertainty-Aware Prompt
Wang, Y., Tang, X., Ma, J., Zhang, X., Liu, F., & Jiao, L. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3458898 · Cited by 34
Cross-Modal Retrieval Augmentation for Multi-Modal Classification
Gur, S., Neverova, N., Stauffer, C., Lim, S.-N., Kiela, D., & Reiter, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.11 · Cited by 29
Knowledge Perceived Multi-modal Pretraining in E-commerce
Zhu, Y., Zhao, H., Zhang, W., Ye, G., Chen, H., Zhang, N., & Chen, H. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475648 · Cited by 30
A Deep Attentive Multimodal Learning Approach for Disaster Identification From Social Media Posts
Hossain, E., Hoque, M. M., Hoque, E., & Islam, M. S. (2022) · IEEE Access · doi:10.1109/access.2022.3170897 · Cited by 29
M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining
Dong, X. M., Zhan, X., Wu, Y., Wei, Y., Kampffmeyer, M., Wei, X.-Y., Lu, M., Wang, Y., & Liang, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02057 · Cited by 36
What Is a Multi-Modal Knowledge Graph: A Survey
Peng, J., Hu, X., Huang, W., & Yang, J. K. (2023) · Big Data Research · doi:10.1016/j.bdr.2023.100380 · Cited by 28
Fully and Weakly Supervised Referring Expression Segmentation With End-to-End Learning
Li, H., Sun, M.-J., Xiao, J., Lim, E. G., & Zhao, Y. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3263468 · Cited by 28
MusCaps: Generating Captions for Music Audio
Manco, I., Benetos, E., Quinton, E., & Fazekas, G. (2021) · 2021 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533461 · Cited by 36
Exploring the Application of Large-Scale Pre-Trained Models on Adverse Weather Removal
Tan, Z., Wu, Y., Liu, Q., Chu, Q., Lü, L., Ye, J., & Yu, N. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3368961 · Cited by 33
Learning to Answer Visual Questions From Web Videos
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3173208 · Cited by 36
Joint Answering and Explanation for Visual Commonsense Reasoning
Li, Z., Guo, Y., Wang, K., Wei, Y., Nie, L., & Kankanhalli, M. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3286259 · Cited by 29
Ilinykh, N., & Dobnik, S. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.767971 · Cited by 26
Visual Relationship Detection With Visual-Linguistic Knowledge From Multimodal Representations
Chiou, M.-J., Zimmermann, R., & Feng, J. (2021) · IEEE Access · doi:10.1109/access.2021.3069041 · Cited by 28
Discrete and continuous representations and processing in deep learning: Looking forward
Cartuyvels, R., Spinks, G., & Moens, M. (2021) · AI Open · doi:10.1016/j.aiopen.2021.07.002 · Cited by 32
Achieving Human Parity on Visual Question Answering
Yan, M., Xu, H., Li, C., Tian, J., Bi, B., Wang, W., Xu, X., Zhang, J., Huang, S., Huang, F., et al. (2022) · ACM Transactions on Information Systems · doi:10.1145/3572833 · Cited by 28
Distilled Dual-Encoder Model for Vision-Language Understanding
Wang, Z., Wang, W., Zhu, H., Liu, M., Qin, B., & Wei, F. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.608 · Cited by 34
Self-citationVideo-Grounded Dialogues with Pretrained Generation Language Models
Le, H., & Hoi, S. C. H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.518 · Cited by 25
UFC-BERT: Unifying Multi-Modal Controls for Conditional Image SynthesisPreprint
Zhang, Z., Ma, J., Zhou, C., Men, R., Li, Z., Ding, M., Tang, J., Zhou, J., & Yang, H. (2021) · arXiv (Cornell University) · Cited by 26
LLaViLo: Boosting Video Moment Retrieval via Adapter-Based Multimodal Modeling
Ma, K., Zang, X., Feng, Z., Fang, H., Ban, C., Wei, Y., He, Z., Li, Y., & Sun, H. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00297 · Cited by 26
How Transferable are Reasoning Patterns in VQA?
Kervadec, C., Jaunet, T., Antipov, G., Baccouche, M., Vuillemot, R., & Wolf, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00419 · Cited by 25
Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration
Liang, X., Zhu, F., Lingling, L., Xu, H., & Liang, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.332 · Cited by 33
Representing Context in FrameNet: A Multidimensional, Multimodal Approach
Torrent, T. T., Matos, E. E. D. S., Belcavello, F., Viridiano, M., Gamonal, M. A., da Costa, A. D., & Marim, M. C. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.838441 · Cited by 31
Unsupervised Domain Adaption Harnessing Vision-Language Pre-Training
Zhou, W., & Zhou, Z. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3391304 · Cited by 26
Phan, V. M. H., Xie, Y., Qi, Y., Liu, L., Liu, L., Zhang, B., Liao, Z., Wu, Q., To, M., & Verjans, J. W. H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01092 · Cited by 26
A Fine-Grained Network for Joint Multimodal Entity-Relation Extraction
Li, Y., Cai, Y., Xu, J., Li, Q., & Wang, T. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3485107 · Cited by 27
Gómez-Pérez, J. M., & Ortega, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.441 · Cited by 27
Learning to Represent Image and Text with Denotation Graph
Zhang, B., Hu, H., Jain, V., Ie, E., & Sha, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.60 · Cited by 23
Declaration-based Prompt Tuning for Visual Question Answering
Liu, Y., Wei, W., Peng, D., & Zhu, F. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/453 · Cited by 29
CMA-CLIP: Cross-Modality Attention Clip for Text-Image Classification
Fu, J., Xu, S., Liu, H., Liu, Y., Xie, N., Wang, C.-C., Liu, J., Sun, Y., & Wang, B. K. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897323 · Cited by 28
Unifying Event Detection and Captioning as Sequence Generation via Pre-training
Zhang, Q., Song, Y., & Jin, Q. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_21 · Cited by 24
Dai, Y., Zou, B., Zhu, C., Li, Y., Chen, Z., Ji, Z., Kui, X., & Zhang, W. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107396 · Cited by 24
Cross‐Modal Graph Contrastive Learning with Cellular Images
Zheng, S., Rao, J., Zhang, J., Zhou, L., Xie, J., Cohen, E., Lu, W., Li, C., & Yang, Y. (2024) · Advanced Science · doi:10.1002/advs.202404845 · Cited by 23
MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question AnsweringPreprint
Khan, A. U., Mazaheri, A., Lobo, N. D. V., & Shah, M. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14095 · Cited by 36
Masking-Based Cross-Modal Remote Sensing Image–Text Retrieval via Dynamic Contrastive Learning
Zhao, Z., Miao, X., He, C., Hu, J., Min, B., Gao, Y., Liu, Y., & Pharksuwan, K. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3406897 · Cited by 24
Inamdar, M. A., Gudigar, A., Raghavendra, U., Salvi, M., Aman, R. R. A. B. R., Gowdh, N. F. M., Ahir, I. A. B. M., Kamaruddin, M. S. B., Kadir, K. A. A., Molinari, F., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3550344 · Cited by 25
An Embarrassingly Simple Model for Dialogue Relation Extraction
Xue, F., Sun, A., Zhang, H., Ni, J., & Chng, E. S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747486 · Cited by 27
Findings of the WOAH 5 Shared Task on Fine Grained Hateful Memes Detection
Mathias, L., Nie, S., Davani, A. M., Kiela, D., Prabhakaran, V., Vidgen, B., & Talat, Z. (2021) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2021.woah-1.21 · Cited by 27
VLDeformer: Vision–Language Decomposed Transformer for fast cross-modal retrieval
Zhang, L., Wu, H., Chen, Q., Deng, Y., Siebert, J. I., Li, Z., Han, Y., Kong, D., & Cao, Z. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109316 · Cited by 25
Non-Contrastive Learning Meets Language-Image Pre-Training
Zhou, J., Li, D., Gan, Z., Wang, L., & Wei, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01061 · Cited by 30
Self-citationRetrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models
Pan, J., Lin, Z., Ge, Y., Zhu, X., Zhang, R., Wang, Y., Qiao, Y., & Li, H. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00035 · Cited by 35
Self-citationOne-Stage Object Referring with Gaze Estimation
Chen, J., Zhang, X., Wu, Y., Ghosh, S., Natarajan, P., Chang, S., & Allebach, J. P. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00550 · Cited by 22
OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis
Hu, J., Mathur, L., Liang, P. P., & Morency, L. (2025) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg61629.2025.11099277 · Cited by 40
Detector-Free Weakly Supervised Grounding by Separation
Arbelle, A., Doveh, S., Alfassy, A., Shtok, J., Lev, G., Schwartz, E., Kuehne, H., Levi, H., Sattigeri, P., Panda, R., et al. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00182 · Cited by 25
FL-MSRE: A Few-Shot Learning based Approach to Multimodal Social Relation Extraction
Wan, H., Zhang, M., Du, J., Huang, Z., Yang, Y., & Pan, J. Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17639 · Cited by 22
UIBert: Learning Generic Multimodal Representations for UI Understanding
Bai, C., Zang, X., Xu, Y., Sunkara, S., Rastogi, A., Chen, J., & y Arcas, B. A. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/235 · Cited by 115
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation
Liu, Y., Wu, C., Tseng, S.-Y., Lal, V., He, X., & Duan, N. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.119 · Cited by 27
González, C., Ayobi, N., Hernández, I., Hernández, J. T., Pont-Tuset, J., & Arbeláez, P. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00140 · Cited by 22
Multimodal Sarcasm Target Identification in Tweets
Wang, J., Sun, L., Liu, Y., Shao, M., & Zheng, Z. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.562 · Cited by 26
SiRi: A Simple Selective Retraining Mechanism for Transformer-Based Visual Grounding
Qu, M., Wu, Y., Liu, W., Gong, Q., Liang, X., Russakovsky, O., Zhao, Y., & Wei, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_32 · Cited by 24
Jiang, J., Liu, Z., & Zheng, N. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01858-y · Cited by 25
Multimodal Religiously Hateful Social Media Memes Classification Based on Textual and Image Data
Hamza, A., Javed, A. R., Iqbal, F., Yasin, A., Srivastava, G., Połap, D., Gadekallu, T. R., & Jalil, Z. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3623396 · Cited by 24
Metadata
- Authors
- Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J.
- Affiliations
- University of Science and Technology of China (CN)
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1908.08530
- OpenAlex
- W2969876226
- Cited by
- 950
- References
- 45
- Citations by year
- 2026: 2 · 2025: 47 · 2024: 170 · 2023: 255 · 2022: 194 · 2021: 197 · 2020: 80 · 2019: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- pre-training, multimodal transformer, region of interest features, visual commonsense reasoning, visual question answering, referring expression comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
