LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Deep Residual Learning for Image Recognition

He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works

Attention Is All You Need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works

Adam: A Method for Stochastic Optimization

Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works

MizAR 60 for Mizar 50

Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works

Very Deep Convolutional Networks for Large-Scale Image Recognition

Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works

ImageNet: A large-scale hierarchical image database

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works

Going deeper with convolutions

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works

Microsoft COCO: Common Objects in Context

Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works

Neural Machine Translation by Jointly Learning to Align and Translate

Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works

Deep Contextualized Word Representations

Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works

VQA: Visual Question Answering

Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works

Film: Visual reasoning with a general conditioning layer

Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490 · see citing works

Bidirectional Attention Flow for Machine ComprehensionPreprint

Seo, M., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01603 · Cited by 2,322 · see citing works

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287 · see citing works

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · see citing works

Untitled work

(2018) · International Conference on Learning Representations · Cited by 1,621 · see citing works

Cross-lingual Language Model Pretraining

Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586 · see citing works

Hierarchical Question-Image Co-Attention for Visual Question AnsweringPreprint

Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,303 · see citing works

VideoBERT: A Joint Model for Video and Language Representation Learning

Sun, C., Myers, A., Vondrick, C., Murphy, K. P., & Schmid, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,212 · see citing works

VisualBERT: A Simple and Performant Baseline for Vision and Language

Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · see citing works

Deep Modular Co-Attention Networks for Visual Question Answering

Zhou, Y., Yu, J., Cui, Y., Tao, D., & Tian, Q. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00644 · Cited by 1,013 · see citing works

Visual7W: Grounded Question Answering in Images

Zhu, Y., Groth, O., Bernstein, M., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.540 · Cited by 794 · cited 2× · see citing works

Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear UnitsPreprint

Hendrycks, D., & Gimpel, K. (2016) · arXiv (Cornell University) · Cited by 753 · see citing works

Gated Feedback Recurrent Neural Networks

Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.02367 · Cited by 583 · see citing works

VQA: Visual Question Answering: www.visualqa.org

Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570 · see citing works

Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering

Zhou, Y., Yu, J., Xiang, C., Jianping Fan, B., & Tao, D. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2817340 · Cited by 536 · see citing works

Learning to Reason: End-to-End Module Networks for Visual Question Answering

Hu, R., Andreas, J., Rohrbach, M., Darrell, T., & Saenko, K. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.93 · Cited by 523 · see citing works

A Corpus for Reasoning about Natural Language Grounded in Photographs

Suhr, A., Zhou, S., Zhang, A. Q., Zhang, I., Bai, H., & Artzi, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1644 · Cited by 487 · see citing works

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487 · see citing works

Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering

Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S. C. H., Wang, X., & Li, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00680 · Cited by 415 · see citing works

Cycle-Consistency for Robust Visual Question Answering

Shah, M., Chen, X., Rohrbach, M., & Parikh, D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00681 · Cited by 165 · see citing works

Pythia v0.1: the Winning Entry to the VQA Challenge 2018Preprint

Jiang, Y. L., Natarajan, V., Chen, X., Rohrbach, M., Batra, D., & Parikh, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.09956 · Cited by 150 · see citing works

VideoBERT: A Joint Model for Video and Language Representation LearningPreprint

Sun, C., Myers, A., Vondrick, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01766 · Cited by 148 · see citing works

Multi-Modality Latent Interaction Network for Visual Question Answering

Gao, P., You, H., Zhang, Z., Wang, X., & Li, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00592 · Cited by 84 · see citing works

GQA: a new dataset for compositional question answering over real-world images.Preprint

Hudson, D. A., & Manning, C. D. (2019) · arXiv (Cornell University) · Cited by 82 · see citing works

Multi-modality Latent Interaction Network for Visual Question AnsweringPreprint

Gao, P., You, H., Zhang, Z., Wang, X., & Li, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04289 · Cited by 55 · see citing works

Multimodal Unified Attention Networks for Vision-and-Language InteractionsPreprint

Zhou, Y., Cui, Y., Yu, J., Tao, D., & Tian, Q. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04107 · Cited by 43 · see citing works

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805

On the Opportunities and Risks of Foundation Models

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works

Pre-Trained Image Processing Transformer

Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01212 · Cited by 2,308 · see citing works

UNITER: UNiversal Image-TExt Representation Learning

Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · see citing works

Pre-trained models for natural language processing: A survey

Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · see citing works

Multimodal learning with transformers: A survey

Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works

Pre-trained models: Past, present and future

Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994 · see citing works

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950 · see citing works

Unified Vision-Language Pre-Training for Image Captioning and VQA

Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7005 · Cited by 864 · see citing works

Transformer Interpretability Beyond Attention Visualization

Chefer, H., Gur, S., & Wolf, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00084 · Cited by 903 · see citing works

Flamingo: a visual language model for few-shot learning

Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · see citing works

Objaverse: A Universe of Annotated 3D Objects

Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsanit, K., Kembhavi, A., & Farhadi, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01263 · Cited by 725 · see citing works

Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks

Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01838 · Cited by 561 · see citing works

Multimodal intelligence: Representation learning, information fusion, and applications

Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511 · see citing works

Deep multimodal data fusion

Zhao, F., Zhang, C., & Geng, B. (2024) · ACM Computing Surveys · doi:10.1145/3649447 · Cited by 480 · see citing works

MTEB: Massive Text Embedding Benchmark

Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.148 · Cited by 548 · see citing works

MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429 · see citing works

Understanding Robustness of Transformers for Image Classification

Bhojanapalli, S., Chakrabarti, A., Glasner, D., Li, D., Unterthiner, T., & Veit, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01007 · Cited by 440 · see citing works

HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

Li, L., Chen, Y.-C., Cheng, Y., Gan, Z., Yu, L., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.161 · Cited by 408 · see citing works

Large language models (LLMs): survey, technical frameworks, and future challenges

Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395 · see citing works

VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

Sung, Y.-L., Cho, J., & Bansal, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00516 · Cited by 329 · see citing works

Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers

Chefer, H., Gur, S., & Wolf, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00045 · Cited by 313 · see citing works

Visual Commonsense R-CNN

Wang, T., Huang, J., Zhang, H., & Sun, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01077 · Cited by 291 · see citing works

UniT: Multimodal Multitask Learning with a Unified Transformer

Hu, R., & Singh, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00147 · Cited by 303 · see citing works

A Review of Deep Learning-Based Information Fusion Techniques for Multimodal Medical Image Classification

Li, Y., El Habib Daho, M., Conze, P.-H., Zeghlache, R., Le Boité, H., Tadayoni, R., Cochener, B., Lamard, M., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108635 · Cited by 274 · see citing works

Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training

Hao, W., Li, C., Li, X., Carin, L., & Gao, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01315 · Cited by 264 · see citing works

VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation

Hong, Y., Wu, Q., Qi, Y., Rodriguez-Opazo, C., & Gould, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00169 · Cited by 264 · see citing works

MultiMAE: Multi-modal Multi-task Masked Autoencoders

Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_20 · Cited by 268 · see citing works

Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models

Liu, Z., Rodriguez-Opazo, C., Teney, D., & Gould, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00213 · Cited by 238 · see citing works

CLIPort: What and Where Pathways for Robotic Manipulation

Shridhar, M., Manuelli, L., & Fox, D. (2021) · arXiv · doi:10.48550/arxiv.2109.12098 · Cited by 223 · see citing works

Multimodal Co-learning: Challenges, applications with datasets, recent advances and future directions

Rahate, A., Walambe, R., Ramanna, S., & Kotecha, K. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.003 · Cited by 214 · see citing works

A Review on Explainability in Multimodal Deep Neural Nets

Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web

Majumdar, A., Shrivastava, A., Lee, S., Anderson, P., Parikh, D., & Batra, D. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_16 · Cited by 201 · see citing works

Causal Attention for Vision-Language Tasks

Yang, X., Zhang, H., Qi, G., & Cai, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00972 · Cited by 175 · see citing works

MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning

Lei, J., Wang, L., Shen, Y., Yu, D., Berg, T., & Bansal, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.233 · Cited by 179 · see citing works

Multimodal Attention-Based Deep Learning for Alzheimer’s Disease Diagnosis

Golovanevsky, M., Eickhoff, C., & Singh, R. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac168 · Cited by 165 · see citing works

Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends

Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155 · see citing works

A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics

He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147 · see citing works

Airbert: In-domain Pretraining for Vision-and-Language Navigation

Guhur, P.-L., Tapaswi, M., Chen, S., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00166 · Cited by 139 · see citing works

KAT: A Knowledge Augmented Transformer for Vision-and-Language

Gui, L., Wang, B., Huang, Q., Hauptmann, A., Bisk, Y., & Gao, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.70 · Cited by 130 · see citing works

TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment

Yang, J., Bisk, Y., & Gao, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01136 · Cited by 115 · see citing works

CLIP-Event: Connecting Text and Images with Event Structures

Li, M., Xu, R., Wang, S., Zhou, L., Lin, X., Zhu, C., Zeng, M., Ji, H., & Chang, S.-F. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01593 · Cited by 117 · see citing works

Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers

Hendricks, L. A., Mellor, J., Schneider, R., Alayrac, J.-B., & Nematzadeh, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00385 · Cited by 100 · see citing works

ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments

An, D., Wang, H., Wang, W., Wang, Z., Huang, Y., He, K., & Wang, L. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3386695 · Cited by 103 · see citing works

Object Detection with Transformers: A Review

Shehzadi, T., Hashmi, K. A., Liwicki, M., Stricker, D., & Afzal, M. Z. (2025) · Sensors · doi:10.3390/s25196025 · Cited by 93 · see citing works

E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning

Xu, H., Yan, M., Li, C., Bi, B., Huang, S., Xiao, W., & Huang, F. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.42 · Cited by 84 · see citing works

X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

Cho, J., Lu, J., Schwenk, D., Hajishirzi, H., & Kembhavi, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.707 · Cited by 99 · see citing works

Vision-Language Navigation with Random Environmental Mixup

Liu, C., Zhu, F., Chang, X., Liang, X., Ge, Z., & Shen, Y.-D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00167 · Cited by 80 · see citing works

A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports

Li, Y., Wang, H., & Luo, Y. (2020) · 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313289 · Cited by 81 · see citing works

ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics

Taleb, A., Kirchler, M., Monti, R., & Lippert, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02024 · Cited by 83 · see citing works

GridMM: Grid Memory Map for Vision-and-Language Navigation

Wang, Z., Li, X., Yang, J., Liu, Y., & Jiang, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01432 · Cited by 93 · see citing works

The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigation

Qi, Y., Pan, Z., Hong, Y., Yang, M.-H., van den Hengel, A., & Wu, Q. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00168 · Cited by 80 · see citing works

Learning to Generate Scene Graph from Natural Language Supervision

Zhong, Y., Shi, J., Yang, J., Xu, C., & Li, Y. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00184 · Cited by 73 · see citing works

Attention-Based Multimodal Fusion with Contrast for Robust Clinical Prediction in the Face of Missing Modalities

Liu, J., Capurro, D., Nguyen, A., & Verspoor, K. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104466 · Cited by 66 · see citing works

Probing Image-Language Transformers for Verb Understanding

Hendricks, L. A., & Nematzadeh, A. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.318 · Cited by 63 · see citing works

Neural natural language generation: A survey on multilinguality, multimodality, controllability and learning

Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54 · see citing works

Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think!

Hessel, J., & Lee, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.62 · Cited by 57 · see citing works

Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-Modal Pretraining

Zhan, X., Wu, Y., Dong, X., Wei, Y., Lu, M., Zhang, Y., Xu, H., & Liang, X. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01157 · Cited by 54 · see citing works

Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network

Li, Y., Pan, Y., Yao, T., Chen, J., & Mei, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17034 · Cited by 49 · see citing works

Visual Chain-of-Thought Prompting for Knowledge-Based Visual Reasoning

Chen, Z., Zhou, Q., Shen, Y., Hong, Y., Sun, Z., Gutfreund, D., & Gan, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i2.27888 · Cited by 48 · see citing works

OpenVLA: An Open-Source Vision-Language-Action Model

Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.09246 · Cited by 62 · see citing works

ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding

Guo, Z., Tang, Y., Zhang, R., Wang, D., Wang, Z., Zhao, B., & Li, X. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01410 · Cited by 44 · see citing works

Memobert: Pre-Training Model with Prompt-Based Learning for Multimodal Emotion Recognition

Zhao, J., Li, R., Jin, Q., Wang, X., & Li, H. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746910 · Cited by 39 · see citing works

GilBERT: Generative Vision-Language Pre-Training for Image-Text Retrieval

Hong, W., Ji, K., Liu, J., Wang, J., Chen, J., & Chu, W. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462838 · Cited by 37 · see citing works

Dynamic alignment and fusion of multimodal physiological patterns for stress recognition

Zhang, X., Wei, X., Zhou, Z., Zhao, Q., Zhang, S., Yang, Y., Li, R., & Hu, B. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3290177 · Cited by 38 · see citing works

VisQA: X-raying Vision and Language Reasoning in Transformers

Jaunet, T., Kervadec, C., Vuillemot, R., Antipov, G., Baccouche, M., & Wolf, C. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114683 · Cited by 30 · see citing works

Pre-training in Medical Data: A Survey

Qiu, Y., Lin, F., Chen, W., & Xu, M. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1382-8 · Cited by 30 · see citing works

How Transferable are Reasoning Patterns in VQA?

Kervadec, C., Jaunet, T., Antipov, G., Baccouche, M., Vuillemot, R., & Wolf, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00419 · Cited by 25 · see citing works

Mindstorms in Natural Language-Based Societies of Mind

Zhuge, M., Liu, H., Faccio, F., Ashley, D. R., Csordás, R., Gopalakrishnan, A., Hamdi, A., Hammoud, H. A. A. K., Herrmann, V., Irie, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.17066 · Cited by 24

Mindstorms in Natural Language-Based Societies of Mind

Zhuge, M., Liu, H., Faccio, F., Ashley, D. R., Csordás, R., Gopalakrishnan, A., Hamdi, A., Hammoud, H. A. A. K., Herrmann, V., Irie, K., et al. (2025) · Computational Visual Media · doi:10.26599/cvm.2025.9450460 · Cited by 24 · see citing works

Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Grounding

Hwang, M., Jeong, J., Kim, M., Oh, Y., & Oh, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00646 · Cited by 26 · see citing works

Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA

Mo, W., & Liu, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28222 · Cited by 20 · see citing works

A Survey of Multimodal Sarcasm Detection

Farabi, S., Ranasinghe, T., Kanojia, D., Kong, Y., & Zampieri, M. (2024) · Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/887 · Cited by 17 · see citing works

Beyond Medical Imaging - A Review of Multimodal Deep Learning in Radiology

Heiliger, L., Sekuboyina, A., Menze, B., Egger, J., & Kleesiek, J. (2022) · doi:10.36227/techrxiv.19103432 · Cited by 13 · see citing works

A Multi-Modal Fusion Method Based on Higher-Order Orthogonal Iteration Decomposition

Liu, F., Chen, J., Tan, W., & Cai, C. (2021) · Entropy · doi:10.3390/e23101349 · Cited by 13 · see citing works

Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers

Miech, A., Alayrac, J.-B., Laptev, I., Sivic, J., & Zisserman, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00970 · Cited by 149 · see citing works

Multi-stage Pre-training over Simplified Multimodal Pre-training Models

Liu, T., Feng, F., & Wang, X. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.199 · Cited by 11 · see citing works

DeepG2P: fusing multi-modal data to improve crop production

Sharma, S., Partap, A., Balaguer, M. A. D. L., Malvar, S., & Chandra, R. (2022) · arXiv · doi:10.48550/arxiv.2211.05986 · Cited by 11 · see citing works

What BERT Sees: Cross-Modal Transfer for Visual Question Generation

Scialom, T., Bordes, P., Dray, P.-A., Staiano, J., & Gallinari, P. (2020) · Proceedings of the 13th International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.39 · Cited by 9 · see citing works

Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models

Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., & Sadigh, D. (2024) · arXiv · doi:10.48550/arxiv.2402.07865 · Cited by 7

A Preliminary Study on Modality of Chinese

Yun, Z. (2002) · Hanyu xuexi

About the Teaching Strategies of Modality in the Classroom

Dilgam, A. G. (2016) · Theory and Practice in Language Studies · doi:10.17507/tpls.0610.05 · Cited by 1 · see citing works

Brief comment on modality in English

Li-xia, S. (2007) · Journal of Northwest University

Factors Influencing Modality Choice in Multimodal Applications

Naumann, A. B., Wechsung, I., & Möller, S. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-69369-7_5 · Cited by 15 · see citing works

Design and Realization of RS Encoder Based on FPGA

De-ming, M. (2009) · Journal of Projectiles.Rockets.Missiles and Guidance