E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning

Deep Residual Learning for Image Recognition

He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works

Attention Is All You Need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works

Microsoft COCO: Common Objects in Context

Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works

Rethinking the Inception Architecture for Computer Vision

Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works

End-to-End Object Detection with Transformers

Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works

Decoupled Weight Decay Regularization

Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works

VQA: Visual Question Answering

Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · see citing works

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · see citing works

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · see citing works

UNITER: UNiversal Image-TExt Representation Learning

Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · see citing works

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Li, X., Yin, X. P., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,736 · see citing works

Microsoft COCO Captions: Data Collection and Evaluation Server

Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · see citing works

Stacked Cross Attention for Image-Text Matching

Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402 · see citing works

VisualBERT: A Simple and Performant Baseline for Vision and Language

Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · see citing works

Circle Loss: A Unified Perspective of Pair Similarity Optimization

Sun, Y., Cheng, C., Zhang, Y., Zhang, C., Zheng, L., Wang, Z., & Wei, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00643 · Cited by 1,034 · see citing works

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950 · see citing works

Unified Vision-Language Pre-Training for Image Captioning and VQA

Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7005 · Cited by 864 · see citing works

MASS: Masked Sequence to Sequence Pre-training for Language GenerationPreprint

Song, K., Tan, X., Qin, T., Lu, J., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02450 · Cited by 834 · see citing works

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training

Li, G., Duan, N., Fang, Y., Gong, M., & Jiang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6795 · Cited by 795 · see citing works

End-to-End People Detection in Crowded Scenes

Stewart, R., Andriluka, M., & Ng, A. Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.255 · Cited by 539 · see citing works

A Corpus for Reasoning about Natural Language Grounded in Photographs

Suhr, A., Zhou, S., Zhang, A. Q., Zhang, I., Bai, H., & Artzi, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1644 · Cited by 487 · see citing works

12-in-1: Multi-Task Vision and Language Representation Learning

Lu, J., Goswami, V., Rohrbach, M., Parikh, D., & Lee, S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01045 · Cited by 410 · see citing works

In Defense of Grid Features for Visual Question Answering

Jiang, H., Misra, I., Rohrbach, M., Miller, E. G., & Chen, X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01028 · Cited by 379 · see citing works

VirTex: Learning Visual Representations from Textual Annotations

Desai, K., & Johnson, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01101 · Cited by 350 · see citing works

Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal TransformersPreprint

Huang, Z., Zeng, Z., Liu, B., Fu, D., & Fu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00849 · Cited by 344 · see citing works

StructBERT: Incorporating Language Structures into Pre-training for Deep Language UnderstandingPreprint

Wang, W., Bi, B., Yan, M., Wu, C., Bao, Z., Xia, J., Peng, L., & Si, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04577 · Cited by 243 · see citing works

ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene GraphPreprint

Yu, F. R., Tang, J., Yin, W., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16934 · Cited by 106 · see citing works

VinVL: Making Visual Representations Matter in Vision-Language ModelsPreprint

Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., & Gao, J. (2021) · arXiv (Cornell University) · Cited by 87 · see citing works

PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation

Bi, B., Li, C., Wu, C. H., Yan, M., Wang, W., Huang, S., Huang, F., & Si, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.700 · Cited by 64 · see citing works

SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple LevelsPreprint

Li, C., Yan, M., Xu, H., Luo, F., Wang, W., Bi, B., & Huang, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.07829 · Cited by 22 · see citing works

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805

Multimodal learning with transformers: A survey

Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works

SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionPreprint

Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., & Cao, Y. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10904 · Cited by 517 · see citing works

X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval

Ma, Y., Xu, G., Sun, X., Yan, M., Zhang, J., & Ji, R. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547910 · Cited by 339 · see citing works

A generalist vision-language foundation model for diverse biomedical tasks

Zhang, K., Zhou, R., Adhikarla, E., Yan, Z., Liu, Y., Yu, J., Liu, Z., Chen, X., Davison, B. D., Ren, H., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03185-2 · Cited by 333 · see citing works

VLP: A Survey on Vision-language Pre-training

Chen, F., Zhang, D., Han, M., Chen, X.-Y., Shi, J. C., Xu, S., & Xu, B. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1369-5 · Cited by 295 · see citing works

MultiMAE: Multi-modal Multi-task Masked Autoencoders

Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_20 · Cited by 268 · see citing works

Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey

Wang, X., Chen, G., Qian, G., Gao, P., Wei, X.-Y., Wang, Y., Tian, Y., & Gao, W. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1410-8 · Cited by 245 · see citing works

mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections

Li, C., Xu, H., Tian, J., Wang, W., Yan, M., Bi, B., Ye, J., Chen, H., Xu, G., Cao, Z., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.488 · Cited by 179 · see citing works

MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining

Dong, X., Bao, J., Zheng, Y., Zhang, T., Chen, D., Yang, H., Zeng, M., Zhang, W., Yuan, L., Chen, D., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01058 · Cited by 177 · see citing works

Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends

Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155 · see citing works

GRIT: Faster and Better Image Captioning Transformer Using Dual Visual Features

Nguyen, V.-Q., Suganuma, M., & Okatani, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_10 · Cited by 152 · see citing works

Multimodal Foundation Models: From Specialists to General-Purpose Assistants

Li, C., Gan, Z., Yang, Z., Yang, J., Li, L., Wang, L., & Gao, J. (2024) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000110 · Cited by 225 · see citing works

Injecting Semantic Concepts into End-to-End Image Captioning

Fang, Z., Wang, J., Hu, X., Liang, L., Gan, Z., Wang, L., Yang, Y., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01748 · Cited by 141 · see citing works

Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual ConceptsPreprint

Zeng, Y., Zhang, X., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08276 · Cited by 153 · see citing works

Image-text Retrieval: A Survey on Recent Research and Development

Cao, M., Li, S., Li, J., Nie, L., & Zhang, M. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/759 · Cited by 125 · see citing works

MixGen: A New Multi-Modal Data Augmentation

Hao, X., Zhu, Y., Appalaraju, S., Zhang, A., Zhang, W., Li, B., & Li, M. (2023) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw58289.2023.00042 · Cited by 101 · see citing works

UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling

Yang, Z., Gan, Z., Wang, J., Hu, X., Ahmed, F., Liu, Z., Lu, Y., & Wang, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_30 · Cited by 83 · see citing works

X2-VLM: All-in-One Pre-Trained Model for Vision-Language Tasks

Zeng, Y., Zhang, X., Li, H., Wang, J., Zhang, J., & Zhou, W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3339661 · Cited by 91 · see citing works

Named Entity and Relation Extraction with Multi-Modal Retrieval

Wang, X., Cai, J., Jiang, Y., Xie, P., Tu, K., & Lü, W. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.437 · Cited by 54 · see citing works

MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model

Ji, Y., Wang, J., Gong, Y., Zhang, L., Zhu, Y., Wang, H., Zhang, J., Sakai, T., & Yang, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02228 · Cited by 48 · see citing works

FashionViL: Fashion-Focused Vision-and-Language Representation Learning

Xiao, H. M., Yu, L., Zhu, X., Zhang, L., Song, Y.-Z., & Xiang, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_37 · Cited by 47 · see citing works

Controllable Image Captioning via Prompting

Wang, N., Xie, J., Wu, J., Jia, M., & Li, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25360 · Cited by 47 · see citing works

Unified multimodal fusion transformer for few shot object detection for remote sensing images

Azeem, A., Li, Z., Siddique, A., Zhang, Y., & Zhou, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102508 · Cited by 39 · see citing works

You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model

Tang, S., Wang, Y., Kong, Z., Zhang, T., Li, Y., Ding, C., Wang, Y., Liang, Y., & Xu, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01038 · Cited by 37 · see citing works

Evolving to multi-modal knowledge graphs for engineering design: state-of-the-art and future challenges

Pan, X., Li, X., Li, Q., Hu, Z., & Bao, J. (2024) · Journal of Engineering Design · doi:10.1080/09544828.2023.2301230 · Cited by 39 · see citing works

Webly Supervised Concept Expansion for General Purpose Vision Models

Kamath, A., Clark, C., Gupta, T., Kolve, E., Hoiem, D., & Kembhavi, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_38 · Cited by 35 · see citing works

Efficient Image Captioning for Edge Devices

Wang, N., Xie, J., Luo, H., Cheng, Q., Wu, J., Jia, M., & Li, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25359 · Cited by 34 · see citing works

Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture

Gupta, T., Kamath, A., Kembhavi, A., & Hoiem, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01591 · Cited by 54 · see citing works

See and Learn More: Dense Caption-Aware Representation for Visual Question Answering

Bi, Y., Jiang, H., Hu, Y., Sun, Y., & Yin, B. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3291379 · Cited by 34 · see citing works

MITER: Medical Image–TExt joint adaptive pretRaining with multi-level contrastive learning

Shu, C., Zhu, Y., Tang, X., Xiao, J., Chen, Y., Li, X., Zhang, Q., & Lu, Z. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121526 · Cited by 34 · see citing works

Transforming Visual Scene Graphs to Image Captions

Yang, X., Peng, J., Wang, Z., Xu, H., Ye, Q., Li, C., Huang, S., Huang, F., Li, Z., & Zhang, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.694 · Cited by 32 · see citing works

Towards Language-Guided Visual Recognition via Dynamic Convolutions

Luo, G., Zhou, Y., Sun, X., Wu, Y., Gao, Y., & Ji, R. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01871-1 · Cited by 29 · see citing works

EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning

Wang, T., Zhou, W., Zeng, Y., & Zhang, X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.873 · Cited by 38 · see citing works

KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation

Liu, Y., Wu, C., Tseng, S.-Y., Lal, V., He, X., & Duan, N. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.119 · Cited by 24 · see citing works

DRESS : Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback

Chen, Y., Sikka, K., Cogswell, M., Ji, H., & Divakaran, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01350 · Cited by 29 · see citing works

MVPTR

Li, Z., Fan, Z., Tou, H., Chen, J., Wei, Z., & Huang, X. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548341 · Cited by 16 · see citing works

FindIt: Generalized Localization with Natural Language Queries

Kuo, W., Bertsch, F., Li, W., Piergiovanni, A., Saffar, M. H., & Angelova, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_29 · Cited by 14 · see citing works

Crossing the Format Boundary of Text and Boxes: Towards Unified Vision-Language ModelingPreprint

Yang, Z., Gan, Z., Wang, J., Hu, X., Ahmed, F., Liu, Z., Lu, Y., & Wang, L. (2021) · arXiv (Cornell University) · Cited by 13 · see citing works

MAGVLT: Masked Generative Vision-and-Language Transformer

Kim, S., Jo, D., Lee, D., & Kim, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02235 · Cited by 12 · see citing works

HybridPrompt: Bridging Language Models and Human Priors in Prompt Tuning for Visual Question Answering

Ma, Z., Yu, Z., Li, J., & Li, G. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26569 · Cited by 12 · see citing works

Transformers in speech processing: Overcoming challenges and paving the future

Latif, S., Zaidi, S. A. M., Cuayáhuitl, H., Shamshad, F., Shoukat, M., Usama, M., & Qadir, J. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100768 · Cited by 68 · see citing works

Unified Multimodal Model with Unlikelihood Training for Visual Dialog

Wang, Z., Wang, J., & Jiang, C. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547974 · Cited by 10 · see citing works

Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation

Jiang, C., Ye, W., Xu, H., Huang, S., Huang, F., & Zhang, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.819 · Cited by 10 · see citing works

Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learning

Ji, Y., Tu, R.-C., Jiang, J., Kong, W., Cai, C., Zhao, W., Wang, H., Yang, Y., & Liu, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00656 · Cited by 10 · see citing works

InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation

Hu, A., Chen, S., Zhang, L., & Jin, Q. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.178 · Cited by 13 · see citing works

BUS : Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarization

Jiang, C., Xu, H., Ye, W., Ye, Q., Li, C., Yan, M., Bi, B., Zhang, S., Huang, F., & Huang, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00271 · Cited by 8 · see citing works

Align vision-language semantics by multi-task learning for multi-modal summarization

Cui, C., Liang, X., Wu, S., & Li, Z. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09908-3 · Cited by 8 · see citing works

Sketch Storytelling

Zhou, Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746558 · Cited by 8 · see citing works

End-to-End Unsupervised Vision-and-Language Pre-training with Referring Expression Matching

Chen, C., Li, P., Sun, M., & Liu, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.742 · Cited by 8 · see citing works

A multimodal dialogue system for improving user satisfaction via knowledge-enriched response and image recommendation

Wang, J., Li, H., Wang, L., & Wu, C. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08409-z · Cited by 7 · see citing works

TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection

Jiang, C., Xu, H., Li, C., Yan, M., Ye, W., Zhang, S., Bi, B., & Huang, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.273 · Cited by 10 · see citing works

CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training

Ma, Z., Li, J., Li, G., & Huang, K. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548292 · Cited by 9 · see citing works

Transformer based Multitask Learning for Image Captioning and Object Detection

Basak, D., Srijith, P. K., & Desarkar, M. S. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2253-2_21 · Cited by 6 · see citing works

Enhancing Vision-Language Pre-Training with Rich Supervisions

Gao, Y., Shi, K., Zhu, P., Belval, E., Nuriel, O., Appalaraju, S., Ghadar, S., Tu, Z., Mahadevan, V., & Soatto, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01280 · Cited by 6 · see citing works

Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets

Cornia, M., Baraldi, L., Fiameni, G., & Cucchiara, R. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01949-w · Cited by 20 · see citing works

Global and Local Semantic Completion Learning for Vision-Language Pre-Training

Tu, R.-C., Ji, Y., Jiang, J., Kong, W., Cai, C., Zhao, W., Wang, H., Yang, Y., & Liu, W. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3596394 · Cited by 7 · see citing works

A Survey of Full-Cycle Cross-Modal Retrieval: From a Representation Learning Perspective

Wang, S., Zhu, L., Shi, L., Mo, H., & Tan, S. (2023) · Applied Sciences · doi:10.3390/app13074571 · Cited by 5 · see citing works

Untitled work

(2023) · doi:10.18653/v1/2023.emnlp-main.824 · Cited by 4 · see citing works

Knowledge-Embedded Mutual Guidance for Visual Reasoning

Zheng, W., Yan, L., Chen, L., Li, Q., & Wang, F. (2023) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2023.3310892 · Cited by 3 · see citing works

TiMix: Text-Aware Image Mixing for Effective Vision-Language Pre-training

Jiang, C., Ye, W., Xu, H., Ye, Q., Yan, M., Zhang, J., & Zhang, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i3.28025 · Cited by 3 · see citing works

Let Me Show You Step by Step: An Interpretable Graph Routing Network for Knowledge-based Visual Question Answering

Wang, D., Hu, L., Hao, R., Shao, Y., Lv, X., Nie, L., & Li, J. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657790 · Cited by 3 · see citing works

Fine-Grained Scene Image Classification with Modality-Agnostic Adapter

Wang, Y., Zhou, Z., Du, X., Wu, X., Zheng, Y., & Jin, C. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10687875 · Cited by 3 · see citing works

Modeling Multimodal Uncertainties via Probability Distribution Encoders Included Vision-Language Models

Wang, J., Ji, Y., Zhang, Y., Zhu, Y., & Sakai, T. (2023) · IEEE Access · doi:10.1109/access.2023.3347192 · Cited by 2 · see citing works

A Review of Multimodal Sentiment Analysis in Online Public Opinion Monitoring

Liu, S., & Li, T. (2026) · Informatics · doi:10.3390/informatics13010010 · Cited by 1 · see citing works

Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs

Li, Y., Liu, Z., Hu, B., Wang, W. W., Ding, Y., Cao, X., & Zhang, M. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3649356 · Cited by 4 · see citing works

Strategies for improving the accuracy of English translation through multimodal fusion deep learning

Lan, S. H. E. N., Bai, Y., & Duan, J. (2026) · Proceedings of the Indian National Science Academy · doi:10.1007/s43538-025-00685-8 · Cited by 1 · see citing works

Improving Image Captioning with Feature Filtering and Injection

Guo, M.-H., Chen, Q., Fang, X., Bao, J., & Xiang, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44210-0_30 · Cited by 1 · see citing works

Text-Image Generation Using Visual-Language Transformer Parallel Decoding

Fang, P., & Wang, F. (2024) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp62122.2024.10743787 · Cited by 1 · see citing works

Shifted Window Fourier Transform and Retention for Image Captioning

Hu, J. C., Cavicchioli, R., & Capotondi, A. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6596-9_12 · Cited by 1 · see citing works

Vision-and-Language Pretraining: Methods, Applications, and Future ChallengesPreprint

Nguyen, T., Nguyen, C.-D. T., Wu, X., Ng, S.-K., & Luu, A.-T. (2023) · Research Square · doi:10.21203/rs.3.rs-3120051/v1 · Cited by 1 · see citing works

Towards Making Transformer-Based Language Models Learn How Children Learn

Mahdy, Y. (2022) · doi:10.18122/td.1975.boisestate

Cross-modal representation learning and generation

Liu, H., Chen, J., Li, L., Bao, B., Li, Z., Liu, J., & Nie, L. (2023) · Journal of Image and Graphics · doi:10.11834/jig.230035 · Cited by 1 · see citing works

Learning Unified Video-Language Representations via Joint Modeling and Contrastive Learning for Natural Language Video Localization

Cui, C., Liang, X., Wu, S., & Li, Z. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191104

Recombination Samples Training for Robust Natural Language Visual Reasoning

Jiang, Y., Zhao, Y., & Bao, B. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00103

Uncertainty Modeling in Multimodal Understanding and Generation via Probability Distribution EncodersPreprint

Wang, J. (2023) · doi:10.36227/techrxiv.24264391

Uncertainty Modeling in Multimodal Understanding and Generation via Probability Distribution EncodersPreprint

Wang, J. (2023) · doi:10.36227/techrxiv.24264391.v1

M33D: Learning 3D priors using Multi-Modal Masked Autoencoders for 2D image and video understanding

Jamal, M. A., & Mohareri, O. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00253

Cross-modal image text retrieval based on cross-optimization and joint strategy

Han, N., Zheng, W., Hu, Y., Kang, P., Zhang, Y., Yang, A., & Zhang, L. (2024) · International Conference on Image, Signal Processing and Pattern Recognition (ISPP) · doi:10.1117/12.3034096 · Cited by 1 · see citing works

Capture Concept Through Comparison: Vision-and-Language Representation Learning with Intrinsic Information Mining

Song, Y.-Z., Chen, Y.-S., Lin, T.-L., Liu, B., Fu, J., & Shuai, H.-H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0908-6_13

Graph Enhanced Cross-Modal Retrieval Based on Visual-Language Knowledge Distillation

Leng, Y., Zhang, H., Liu, L., & Liu, D. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-7008-6_7

AMFNVD: Addressing Two Kinds of Discrepancy Problems for Fake News Detection on Short Video Platforms

Cheng, Zhang, Yang, Y., & Liu, Y. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4100-3_27

A Language-Image Pre-training Model Based on Context Optimization and Region of Interest

Jin, R., Jin, T., Li, Z., Wu, T., Wang, Y., Zhu, Q., & Luo, M. (2025) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-025-01061-6

Exploration and Practice of ZiDongTaiChu Multimodal Large Model

Wang, J., & Yang, B. (2026) · Springer eBooks · doi:10.1007/978-981-96-8252-2_15

Cross-Modal Comprehensive Multi-Level Granularity Alignment for Vision-Language Pre-Training

jiang, J., Vukovic, D. B., Cao, J., Xie, H., & Wang, Y. (2026) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3802546

Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions

Li, L. H., You, H., Wang, Z., Zareian, A., Chang, S., & Chang, K.-W. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.420 · Cited by 45 · see citing works

MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models

Monajatipoor, M., Li, L. H., Rouhsedaghat, M., Yang, L. F., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.43 · Cited by 9 · see citing works

Effective End-to-End Vision Language Pretraining With Semantic Visual Loss

Yang, X., Liu, F., & Lin, G. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3237166 · Cited by 14 · see citing works

VL-BEiT: Generative Vision-Language PretrainingPreprint

Bao, H., Wang, W., Li, D., & Wei, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.01127 · Cited by 24 · see citing works

Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference

Wang, S., Wang, D., Zhou, C., Li, Z., Fan, Z., Huang, X., & Wei, Z. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1484 · Cited by 1 · see citing works