Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
Open accessAbstract
Recently, multimodal transformer models have gained popularity because their performance on downstream tasks suggests they learn rich visual-linguistic representations. Focusing on zero-shot image retrieval tasks, we study three important factors that can impact the quality of learned representations: pretraining data, the attention mechanism, and loss functions. By pretraining models on six datasets, we observe that dataset noise and language similarity to our downstream task are important indicators of model performance. Through architectural analysis, we learn that models with a multimodal attention mechanism can outperform deeper models with modality-specific attention mechanisms. Finally, we show that successful contrastive losses used in the self-supervised learning literature do not yield similar performance gains when used in multimodal transformers.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Curating higher-quality image-text datasets may be more important than generating large single-modality datasets, and other aspects of algorithm development and training.
“A study from DeepMind, in fact, suggested that curating higher-quality image-text datasets may be more important than generating large single-modality datasets, and other aspects of algorithm development and training 119”
From Multimodal biomedical AI (2022) · cites this paper as [119]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,037 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Kudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · see citing works
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01316-z · Cited by 2,460 · see citing works
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · see citing works
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · see citing works
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,309 · see citing works
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287 · see citing works
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00982 · Cited by 2,194 · see citing works
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · see citing works
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · see citing works
Tian, Y., Krishnan, D., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_45 · Cited by 1,997 · see citing works
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · see citing works
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X. P., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,736 · see citing works
Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary
Duygulu, P., Barnard, K., Freitas, J., & Forsyth, D. A. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47979-1_7 · Cited by 1,624 · see citing works
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · see citing works
Unifying Visual-Semantic Embeddings with Multimodal Neural Language ModelsPreprint
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,386 · see citing works
Hierarchical Question-Image Co-Attention for Visual Question AnsweringPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,303 · see citing works
VideoBERT: A Joint Model for Video and Language Representation Learning
Sun, C., Myers, A., Vondrick, C., Murphy, K. P., & Schmid, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,212 · see citing works
VisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · see citing works
VSE++: Improving Visual-Semantic Embeddings with Hard NegativesPreprint
Faghri, F., Fleet, D. J., Kiros, J., & Fidler, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05612 · Cited by 1,002 · see citing works
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00272 · Cited by 986 · see citing works
Self-citationIm2Text: Describing Images Using 1 Million Captioned Photographs
Ordóñez, V., Kulkarni, G. A., & Berg, T. L. (2011) · Cited by 879 · see citing works
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
Li, G., Duan, N., Fang, Y., Gong, M., & Jiang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6795 · Cited by 795 · see citing works
Learning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.541 · Cited by 788 · see citing works
Wsabie: Scaling up to large vocabulary image annotation
Weston, J., Samy Bengio, A., & Usunier, N. (2011) · Cited by 659 · see citing works
End-to-End Learning of Visual Representations From Uncurated Instructional Videos
Miech, A., Alayrac, J.-B., Smaira, L., Laptev, I., Šivic, J., & Zisserman, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00990 · Cited by 622 · see citing works
Self-citationLarge Batch Optimization for Deep Learning: Training BERT in 76 minutesPreprint
You, Y., Li, J., Reddi, S. J., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J. W., Keutzer, K., & Hsieh, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.00962 · Cited by 597 · see citing works
A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their Semantics
Gong, Y., Ke, Q., Isard, M., & Lazebnik, S. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0658-4 · Cited by 539 · see citing works
Tian, Y., Krishnan, D., & Isola, P. (2019) · arXiv · doi:10.48550/arxiv.1906.05849 · Cited by 465 · see citing works
Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge
Teney, D., Anderson, P., He, X., & van den Hengel, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00444 · Cited by 398 · see citing works
Designing and Interpreting Probes with Control Tasks
Hewitt, J., & Liang, P. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1275 · Cited by 393 · see citing works
Hierarchical Question-Image Co-Attention for Visual Question Answering
Lu, J. (2024) · TIB Data Manager · doi:10.57702/uhdmfu1r · Cited by 357 · see citing works
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal TransformersPreprint
Huang, Z., Zeng, Z., Liu, B., Fu, D., & Fu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00849 · Cited by 344 · see citing works
Learning a Text-Video Embedding from Incomplete and Heterogeneous DataPreprint
Miech, A., Laptev, I., & Šivic, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.02516 · Cited by 217 · see citing works
Information-Theoretic Probing with Minimum Description Length
Voita, E., & Titov, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.14 · Cited by 197 · see citing works
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives.
Faghri, F., Fleet, D. J., Kiros, J., & Fidler, S. (2018) · Cited by 184 · see citing works
Connecting Vision and Language with Localized Narratives
Pont-Tuset, J., Uijlings, J., Changpinyo, S., Soricut, R., & Ferrari, V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58558-7_38 · Cited by 176 · see citing works
Learning and Evaluating General Linguistic IntelligencePreprint
Yogatama, D., d’Autume, C. D. M., Connor, J. T., Kočiský, T., Chrzanowski, M., Kong, L., Lazaridou, A., Wang, L., Yu, L., Dyer, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11373 · Cited by 171 · see citing works
Zhang, K., & Bowman, S. R. (2018) · doi:10.18653/v1/w18-5448 · Cited by 113 · see citing works
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
Cao, J., Gan, Z., Cheng, Y., Yu, L., Chen, Y.-C., & Liu, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58539-6_34 · Cited by 105 · see citing works
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval
Mithun, N. C., Panda, R., Papalexakis, E. E., & Roy–Chowdhury, A. K. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240712 · Cited by 79 · see citing works
Akula, A., Gella, S., Al-Onaizan, Y., Zhu, S. C., & Reddy, S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.586 · Cited by 45 · see citing works
Are we pretraining it right? Digging deeper into visio-linguistic pretrainingPreprint
Singh, A., Goswami, V., & Parikh, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08744 · Cited by 42 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
4 of the 62 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Acosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365 · see citing works
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BackgroundSelf-citationAn Empirical Study of Training End-to-End Vision-and-Language Transformers
Dou, Z.-Y., Xu, Y., Gan, Z., Wang, J., Wang, S., Wang, L., Zhu, C., Zhang, P., Yuan, L., Peng, N., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01763 · Cited by 366 · see citing works
Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval
Jiang, D., & Ye, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00273 · Cited by 348 · see citing works
Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 306 · see citing works
Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155 · see citing works
A Simple Framework for Open-Vocabulary Segmentation and Detection
Zhang, H., Li, F., Zou, X., Liu, S., Li, C., Yang, J., & Zhang, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00100 · Cited by 149 · see citing works
MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition
Chen, J., & Ho, C. M. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00086 · Cited by 139 · see citing works
A comprehensive survey of large language models and multimodal large language models in medicine
Xiao, H., Zhou, F., Liu, X., Liu, T., Ли, Ж., Liu, X., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102888 · Cited by 130 · see citing works
Self-Supervised Multimodal Learning: A Survey
Zong, Y., Aodha, O. M., & Hospedales, T. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429301 · Cited by 122 · see citing works
Pedestrian-specific Bipartite-aware Similarity Learning for Text-based Person Retrieval
Shen, F., Shu, X., Du, X., & Tang, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612009 · Cited by 97 · see citing works
MissModal: Increasing Robustness to Missing Modality in Multimodal Sentiment Analysis
Lin, R., & Hu, H. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00628 · Cited by 91 · see citing works
Zhang, X., Li, M., Lin, S., Xu, H., & Xiao, G. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3312858 · Cited by 79 · see citing works
Codon language embeddings provide strong signals for use in protein engineering
Outeiral, C., & Deane, C. M. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00791-0 · Cited by 87 · see citing works
BridgeTower: Building Bridges between Encoders in Vision-Language Representation Learning
Xiao, X., Wu, C., Rosenman, S., Lal, V., Che, W., & Duan, N. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26263 · Cited by 75 · see citing works
Artificial intelligence for dementia—Applied models and digital health
Lyall, D. M., Kormilitzin, A., Lancaster, C. L., Sousa, J., Petermann-Rocha, F. E., Buckley, C., Harshfield, E. L., Iveson, M. H., Madan, C. R., Ardle, R. M., et al. (2023) · Alzheimer s & Dementia · doi:10.1002/alz.13391 · Cited by 72 · see citing works
Probing Image-Language Transformers for Verb Understanding
Hendricks, L. A., & Nematzadeh, A. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.318 · Cited by 63 · see citing works
Self-citationMultimodal multi-instance learning for long-term ECG classification
Han, H., Lian, C., Zeng, Z., Xu, B., Zang, J., & Xue, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110555 · Cited by 60 · see citing works
mmMIC: Multi-modal Speech Recognition based on mmWave Radar
Fan, L., Xie, L., Lu, X., Li, Y., Wang, C., & Lu, S. (2023) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom53939.2023.10229085 · Cited by 44 · see citing works
Ding, P., Wang, Y., Zhang, X., Gao, X., Liu, G., & Yu, B. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad231 · Cited by 42 · see citing works
SwinCross: Cross‐modal Swin transformer for head‐and‐neck tumor segmentation in PET/CT images
Li, G. Y., Chen, J., Jang, S., Gong, K., & Li, Q. (2023) · Medical Physics · doi:10.1002/mp.16703 · Cited by 36 · see citing works
Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal TransformersPreprint
Yang, R., Zhang, M., Hansen, N., Xu, H., & Wang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03996 · Cited by 45 · see citing works
Separating Skills and Concepts for Novel Visual Question Answering
Whitehead, S., Wu, H., Ji, H., Feris, R., & Saenko, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00558 · Cited by 33 · see citing works
Zhi-hua, W., Yu, L., Ding, X., Liao, X., & Wang, L. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3287256 · Cited by 27 · see citing works
Telling the What while Pointing to the Where: Multimodal Queries for Image Retrieval
Changpinyo, S., Pont-Tuset, J., Ferrari, V., & Soricut, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01192 · Cited by 25 · see citing works
Distilled Dual-Encoder Model for Vision-Language Understanding
Wang, Z., Wang, W., Zhu, H., Liu, M., Qin, B., & Wei, F. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.608 · Cited by 30 · see citing works
Zhang, J., Ruan, Z., Li, Q., & Zhang, Z. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3325522 · Cited by 21 · see citing works
Tutorial on Multimodal Machine Learning
Morency, L., Liang, P. P., & Zadeh, A. (2022) · doi:10.18653/v1/2022.naacl-tutorials.5 · Cited by 20 · see citing works
Shen, Z., Liu, Z., & Xing, E. P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20053-3_42 · Cited by 19 · see citing works
Data Efficient Masked Language Modeling for Vision and Language
Bitton, Y., Elhadad, M., Stanovsky, G., & Schwartz, R. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.259 · Cited by 18 · see citing works
Yu, X., Elazab, A., Ge, R., Zhu, J., Zhang, L., Jia, G., Wu, Q., Wan, X., Li, L., & Wang, C. (2025) · Neural Networks · doi:10.1016/j.neunet.2024.107096 · Cited by 18 · see citing works
Optical Flow-Aware-Based Multi-Modal Fusion Network for Violence Detection
Xiao, Y., Gao, G., Wang, L., & Lai, H. (2022) · Entropy · doi:10.3390/e24070939 · Cited by 16 · see citing works
Remote Sensing Image-Text Retrieval With Implicit-Explicit Relation Reasoning
Yang, L., Zhou, T., Ma, W., Du, M., Liu, L., Li, F., Zhao, S., & Wang, Y. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3466909 · Cited by 16 · see citing works
Learning visual language models for video understanding
Yang, A. (2023) · theses.fr (ABES) · doi:10.70675/802872a7z8092z4556za1fdz2c7af45f0e72 · Cited by 15 · see citing works
PGVT: Pose-Guided Video Transformer for Fine-Grained Action Recognition
Zhang, H., Leong, M. C., Li, L., & Lin, W. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00651 · Cited by 12 · see citing works
Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation
Pezzelle, S., Takmaz, E., & Fernández, R. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00443 · Cited by 11 · see citing works
PeVL: Pose-Enhanced Vision-Language Model for Fine-Grained Human Action Recognition
Zhang, H., Leong, M. C., Li, L., & Lin, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01784 · Cited by 11 · see citing works
Multimodal transformer for early alarm prediction
Strem, N., Dhami, D. S., Schmidt, B., & Kersting, K. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109643 · Cited by 11 · see citing works
Transformers in speech processing: Overcoming challenges and paving the future
Latif, S., Zaidi, S. A. M., Cuayáhuitl, H., Shamshad, F., Shoukat, M., Usama, M., & Qadir, J. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100768 · Cited by 68 · see citing works
Wang, X., Lian, Q., Qu, P., & Yang, Q. J. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3355758 · Cited by 11 · see citing works
AutoTransition: Learning to Recommend Video Transition Effects
Shen, Y., Zhang, L., Xu, K., & Jin, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19839-7_17 · Cited by 10 · see citing works
Measuring Progress in Fine-grained Vision-and-Language Understanding
Bugliarello, E., Sartran, L., Agrawal, A., Hendricks, L. A., & Nematzadeh, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.87 · Cited by 13 · see citing works
Self-citationFine-grained text-based person re-identification via interlaced cross-attention and LoRA fine-tuning
Hu, M., Zhang, W., Zhou, Q., & Wang, R. (2025) · The Visual Computer · doi:10.1007/s00371-025-03931-8 · Cited by 8 · see citing works
Visual context learning based on cross-modal knowledge for continuous sign language recognition
Liu, K., Hou, Y., Guo, Z., Yin, W., & Ren, Y. C. (2024) · The Visual Computer · doi:10.1007/s00371-024-03643-5 · Cited by 8 · see citing works
Lin, W.-C., Goncalves, L., & Busso, C. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3577190.3614110 · Cited by 7 · see citing works
APT: Alarm Prediction Transformer
Strem, N., Dhami, D. S., Schmidt, B., Klöpper, B., & Kristian Kersting, B. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125521 · Cited by 7 · see citing works
Multimodal Transformer Fusion for Emotion Recognition: A Survey
Belaref, A., & Séguier, R. (2024) · doi:10.1109/icnlp60986.2024.10692953 · Cited by 6 · see citing works
Pooling Transformer for Detection of Risk Events in In-The-Wild Video Ego Data
Mallick, R., Benois‐Pineau, J., Zemmari, A., Yebda, T., Pech, M., Amieva, H., & Middleton, L. E. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956675 · Cited by 6 · see citing works
A Multimodal Deep Neural Network for ECG and PCG Classification With Multimodal Fusion
Han, H., Xiang, M., Lian, C., Liu, D., & Zeng, Z. (2023) · International Conference on Information Science and Technology · doi:10.1109/icist59754.2023.10367180 · Cited by 6 · see citing works
Improving Image Representations via MoCo Pre-training for Multimodal CXR Classification
Serra, F. D., Jacenków, G., Deligianni, F., Dalton, J., & O’Neil, A. Q. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-12053-4_46 · Cited by 5 · see citing works
Late Fusion with Triplet Margin Objective for Multimodal Ideology Prediction and Analysis
Qiu, C., Wu, W. M. C., Zhang, X. F., & Wang, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.659 · Cited by 5 · see citing works
Li, W., Li, X., Li, Z., Yu, J., & Chen, P. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130622 · Cited by 4 · see citing works
Enhancing cross view geo localization through global local quadrant interaction network
Xu, J. X., Junping, Y., Zhang, J., & Tianyan, G. (2025) · Scientific Reports · doi:10.1038/s41598-025-18935-6 · Cited by 4 · see citing works
Multimodal prediction of catheter ablation outcomes in patients with persistent atrial fibrillation
Zhou, Z., Chen, Y., Zhang, F., Liu, H., Karimi, H. R., & Cao, J. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107835 · Cited by 4 · see citing works
ChitroJera: A Regionally Relevant Visual Question Answering Dataset for Bangla
Barua, D. D., Sourove, M. S. U. R., Fahim, M., Haider, F., Shifat, F. T., Adib, M. T. R., Uddin, A. B., Ishmam, M. F., & Alam, M. F. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06078-5_27 · Cited by 4 · see citing works
Vision-Language Pretraining: Current Trends and the Future
Agrawal, A., Teney, D., & Nematzadeh, A. (2022) · doi:10.18653/v1/2022.acl-tutorials.7 · Cited by 4 · see citing works
Self-citationMultimodal Event Classification for Social Media Based on Text-Image-Caption Assisted Alignment
Wang, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3496001 · Cited by 4 · see citing works
Incorporating multi-path risk assessment in transformer-based pedestrian crossing action prediction
Liu, B., Li, M., Feng, R., Zhou, W., & Li, Z. (2025) · Accident Analysis & Prevention · doi:10.1016/j.aap.2025.108002 · Cited by 4 · see citing works
AMFMER: A multimodal full transformer for unifying aesthetic assessment tasks
Qi, J., Su, C., Hu, X., Chen, M., Sun, Y., Dong, Z., Liu, T., & Luo, J. (2025) · Signal Processing Image Communication · doi:10.1016/j.image.2025.117320 · Cited by 4 · see citing works
MEGA: Multimodal Alignment Aggregation and Distillation For Cinematic Video Segmentation
Sadoughi, N., Li, X., Vajpayee, A., Fan, D., Shuai, B., Santos-Villalobos, H., Bhat, V., & Rohith, M. V. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02132 · Cited by 4 · see citing works
TetraCVD: A Temporal-Textual Transformer based Model for Cardiovascular Disease Diagnosis
Lu, K., Zhao, F., Gu, P., Wang, G., Zang, T., & Wang, H. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385761 · Cited by 4 · see citing works
Huang, S., Hong, Z., Wu, B., Liang, J., & Huang, Q. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109933 · Cited by 3 · see citing works
El-Ateif, S., Idri, A., & Fernández‐Alemán, J. L. (2024) · Data Technologies and Applications · doi:10.1108/dta-01-2023-0005 · Cited by 3 · see citing works
Analyzing the Robustness of Vision & Language Models
Shirnin, A., Andreev, N., Potapova, S., & Artemova, E. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3399061 · Cited by 2 · see citing works
Zhu, Y., Shi, X., Wang, L., & Zhang, J. (2025) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-025-00777-5 · Cited by 2 · see citing works
Wang, S., & Guo, K. (2026) · Journal of Bionic Engineering · doi:10.1007/s42235-026-00865-2 · Cited by 2 · see citing works
Prevention of Global Mental Health Crisis with Transformer Neural Networks
Rajagopal, A. R., Nirmala, V., Andrew, J., Arun, M., & Piush, A. (2023) · Intelligent systems reference library · doi:10.1007/978-3-031-12419-8_11 · Cited by 2 · see citing works
Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining
Bugliarello, E., Nematzadeh, A., & Hendricks, L. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.184 · Cited by 2 · see citing works
Self-citationHan, H., Lian, C., Xu, B., Zeng, Z., Alhudhaif, A., & Polat, K. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112372 · Cited by 2 · see citing works
Boosting Generic Visual-Linguistic Representation With Dynamic Contexts
Ma, G., Bai, Y., Zhang, W., Yao, T., Shihada, B., & Mei, T. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3237164 · Cited by 2 · see citing works
Separating Skills and Concepts for Novel Visual Question AnsweringPreprint
Whitehead, S., Wu, H., Ji, H., Feris, R., & Saenko, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09106 · Cited by 2 · see citing works
Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions
Cafagna, M., van Deemter, K., & Albert Gatt, A. (2022) · doi:10.18653/v1/2022.umios-1.6 · Cited by 1 · see citing works
A Multimodal Benchmark and Improved Architecture for Zero Shot Learning
Doshi, K., Garg, A., Uzkent, B., Wang, X., & Omar, M. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00202 · Cited by 1 · see citing works
MultiHeart: Secure and Robust Heartbeat Pattern Recognition in Multimodal Cardiac Monitoring System
Ahmadi, H., Zhang, Y., & Tran, N. H. (2025) · Electronics · doi:10.3390/electronics14153149 · Cited by 1 · see citing works
A Hybrid Transformer Network for Detection of Risk Situations on Multimodal Life-Log Health Data
Mallick, R., Benois‐Pineau, J., Zemmari, A., Pech, M., Yebda, T., Amieva, H., & Middleton, L. E. (2022) · ACM Workshop on Intelligent Cross-Data Analysis and Retrieval (ICDAR) · doi:10.1145/3512731.3534212 · Cited by 1 · see citing works
Vision-and-Language Pretraining: Methods, Applications, and Future ChallengesPreprint
Nguyen, T., Nguyen, C.-D. T., Wu, X., Ng, S.-K., & Luu, A.-T. (2023) · Research Square · doi:10.21203/rs.3.rs-3120051/v1 · Cited by 1 · see citing works
One does not fit all! On the Complementarity of Vision Encoders for Vision and Language Tasks
Geigle, G., Liu, C., Pfeiffer, J., & Gurevych, I. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.9
Amfmer: A Multimodal Full Transformer for Unifying Aesthetic Assessment Tasks ⋆Preprint
Qi, J., Su, C., Hu, X., Chen, M., Sun, Y., Dong, Z.-J., Cao, Y., Liu, T., & Luo, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4608636
Amfmer: A Multimodal Full Transformer for Unifying Aesthetic Assessment TasksPreprint
Qi, J., Su, C., Hu, X., Chen, M., Sun, Y., Dong, Z.-J., Liu, T., & Luo, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4864935
Wu, C., Zhang, Y., Wang, Y., Wang, Q., & Wang, S. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635355
Dual-feature adaptive framework for multimodal disinformation detection
Yan, K., Liang, G., Wang, L., Sun, M., & Zhao, K. (2025) · The Computer Journal · doi:10.1093/comjnl/bxaf024
Dual-Modality Feature Blending: A Channel-Aware Modeling for Multimodal IntegrationPreprint
Bianchi, L., Hsu, L., & Romano, G. (2025) · Preprints.org · doi:10.20944/preprints202504.0350.v1
Cross-Modal Fine-grained Feature Alignment for Text-Based Person Re-identificationPreprint
Lin, Q., Zhang, C., Zhou, X., & Feng, Z. (2025) · Research Square · doi:10.21203/rs.3.rs-6769139/v1
Cross-Feature Hybrid Associative Priori Network for Pulsar Candidate Screening
Luo, W., Xie, X., Jiang, J., Zhou, L., & Hu, Z. (2025) · Sensors · doi:10.3390/s25133963
Explainability of Language Models (XAI and LLM)
Cecco, A. D., & Gianfagna, L. (2025) · Springer eBooks · doi:10.1007/978-3-031-92229-9_8
CanadaFireSat: Towards high-resolution wildfire forecasting with multiple modalities
Porta, H., Dalsasso, E., McCarty, J. L., & Tuia, D. (2026) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2026.05.050
Yao, H. (2025) · IEEE International Conference on Advances in Electrical Engineering and Computer Applications (AEECA) · doi:10.1109/aeeca65693.2025.00133
Zhou, X., Luo, S., Song, L., & Hou, H. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402549
Chaffin, A. (2023) · theses.fr (ABES) · doi:10.70675/5e4cd6f0z8697z4a0cz921dzd8592affa0a9
Explainable Deep Learning for the Application to Multimodal Data
Mallick, R. (2023) · theses.fr (ABES) · doi:10.70675/84d86f53z6799z4da8z9847z16ce10bbfdb5
Study of the multimodal understanding of vision-language transformer models
Salin, E. (2023) · theses.fr (ABES) · doi:10.70675/2beca704z5d3bz421dzbf61z61e00610aaeb
Enhancing Cross-Modal Person Re-Identification through Fine-Grained Feature Fusion
Zhang, B., Liu, P., Wang, H., Luo, M., Dou, Q., & Song, Y. (2026) · International Symposium on Computer Applications and Information Systems (ISCAIS) · doi:10.1109/iscait69154.2026.11477493
Multimodal Fusion of 3D CT and Pathological Images for Gastric Cancer Recurrence Prediction
Cao, L., Tian, M., Li, J., Chen, Z., Wang, X., Zhong, S., Wu, G., Tang, Z., & Yu, J. (2026) · Cancer Management and Research · doi:10.2147/cmar.s563640
Compressed video-driven multimodal modeling and interaction for dynamic expression recognition
Xie, W., Zhang, J., Liang, H., Shen, L., Lai, Z., Song, S., & Yu, Z. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116590
CMF2A: Cross-Modal Fine-grained Feature Alignment for text-based person re-identification
Lin, Q., Zhang, C., Zhou, X., & feng, Z. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116649
- Authors
- Hendricks, L. A., Mellor, J., Schneider, R., Alayrac, J.-B., & Nematzadeh, A.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, United Kingdom. rgschneider@google.com
- Corresponding author
- Hendricks, Lisa Anne
- Mellor, John
- Alayrac, Jean-Baptiste
- Nematzadeh, Aida
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 570-585
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3184369217
- Cited by
- 100
- References
- 62
- Citations by year
- 2026: 7 · 2025: 19 · 2024: 19 · 2023: 29 · 2022: 18 · 2021: 8
- FWCI
- 6.65
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal transformer, zero-shot image retrieval, pre-training data, language similarity, multimodal attention, modality-specific attention, contrastive loss, self-supervised learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
