VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

ImageNet classification with deep convolutional neural networks

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155

Long Short-Term Memory

Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318

Adam: A Method for Stochastic Optimization

Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946

ImageNet: A large-scale hierarchical image database

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333

Gradient-based learning applied to document recognition

Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325

Efficient Estimation of Word Representations in Vector Space

Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568

Language Models are Few-Shot Learners

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372

End-to-End Object Detection with Transformers

Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285

Momentum Contrast for Unsupervised Visual Representation Learning

He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347

On the Opportunities and Risks of Foundation Models

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676

Multimodal biomedical AI

Acosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365

Multimodal learning with transformers: A survey

Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072

Self-supervised learning in medicine and healthcare

Krishnan, R., Rajpurkar, P., & Topol, E. J. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00914-1 · Cited by 605

Deep multimodal data fusion

Zhao, F., Zhang, C., & Geng, B. (2024) · ACM Computing Surveys · doi:10.1145/3649447 · Cited by 480

Attention Bottlenecks for Multimodal Fusion

Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., & Sun, C. (2021) · arXiv · doi:10.48550/arxiv.2107.00135 · Cited by 400

Audioclip: Extending Clip to Image, Text and Audio

Guzhov, A., Raue, F., Hees, J., & Dengel, A. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747631 · Cited by 330

Multiview Transformers for Video Recognition

Yan, S., Xiong, X., Arnab, A., Lu, Z., Zhang, M., Sun, C., & Schmid, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00333 · Cited by 312

A Review of Deep Learning-Based Information Fusion Techniques for Multimodal Medical Image Classification

Li, Y., El Habib Daho, M., Conze, P.-H., Zeghlache, R., Le Boité, H., Tadayoni, R., Cochener, B., Lamard, M., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108635 · Cited by 274

MultiMAE: Multi-modal Multi-task Masked Autoencoders

Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_20 · Cited by 268

BEVT: BERT Pretraining of Video Transformers

Wang, R., Chen, D., Wu, Z., Chen, Y., Dai, X., Liu, M., Jiang, Y.-G., Zhou, L., & Yuan, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01432 · Cited by 234

Multimodal Co-learning: Challenges, applications with datasets, recent advances and future directions

Rahate, A., Walambe, R., Ramanna, S., & Kotecha, K. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.003 · Cited by 214

Are Multimodal Transformers Robust to Missing Modality?

Ma, M., Ren, J., Zhao, L., Testuggine, D., & Peng, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01764 · Cited by 198

When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations

Chen, X., Hsieh, C.-J., & Gong, B. (2021) · arXiv · doi:10.48550/arxiv.2106.01548 · Cited by 128

M3Care: Learning with Missing Modalities in Multimodal Healthcare Data

Zhang, C., Chu, X., Ma, L., Zhu, Y., Wang, Y., Wang, J., & Zhao, J. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539388 · Cited by 102

Contrastive Learning Based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition With Missing Modalities

Liu, R., Zuo, H., Lian, Z., Schuller, B. W., & Li, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3378570 · Cited by 96

COCOA: Cross Modality Contrastive Learning for Sensor Data

Deldari, S., Xue, H., Saeed, A., Smith, D. V., & Salim, F. D. (2022) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3550316 · Cited by 92

XAI-based Clinical Decision Support System: A Systematic Review

Kim, S., Kim, D., Kim, M., Ko, H., & Jeong, O. (2024) · doi:10.20944/preprints202406.0721.v1 · Cited by 72

LLM-FP4: 4-Bit Floating-Point Quantized Transformers

Liu, S.-Y., Liu, Z., Huang, X., Dong, P., & Cheng, K.-T. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.39 · Cited by 50

Decoupling Common and Unique Representations for Multimodal Self-supervised Learning

Wang, Y., Albrecht, C. M., Braham, N. A. A., Liu, C., Xiong, Z., & Zhu, X. X. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73397-0_17 · Cited by 49

Memobert: Pre-Training Model with Prompt-Based Learning for Multimodal Emotion Recognition

Zhao, J., Li, R., Jin, Q., Wang, X., & Li, H. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746910 · Cited by 39

Collective predictive coding hypothesis: symbol emergence as decentralized Bayesian inference

Taniguchi, T. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1353870 · Cited by 28

Untitled work

(2023) · doi:10.1109/cvpr52729.2023.00627 · Cited by 28

Token turing machines

Ryoo, M. S., Gopalakrishnan, K., Kahatapitiya, K., Xiao, T., Rao, K., Stone, A., Lu, Y., Ibarz, J., & Arnab, A. (2022) · arXiv · doi:10.48550/arxiv.2211.09119 · Cited by 21

Identifying progression-specific Alzheimer’s subtypes using multimodal transformer

Machado Reyes, D., Chao, H., Hahn, J., Shen, L., Yan, P., & Initiative, F. T. A. D. N. (2024) · Journal of Personalized Medicine · doi:10.3390/jpm14040421 · Cited by 17

Joint self-supervised and supervised contrastive learning for multimodal MRI data: Towards predicting abnormal neurodevelopment

Li, Z., Li, H., Ralescu, A. L., Dillman, J. R., Altaye, M., Cecil, K. M., Parikh, N. A., & He, L. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102993 · Cited by 10

Lifelong Learning of Large Language Model Based Agents: A Roadmap

Zheng, J., Shi, C., Cai, X., Li, Q., Zhang, D., Li, C., Yu, D., & Ma, Q. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3650546 · Cited by 7

Enhancing Multimodal Model Robustness Under Missing Modalities via Memory-Driven Prompt Learning

Zhao, Y., Xi, W., Fu, X., & Zhao, J. (2025) · Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2025/274 · Cited by 6

Exploring Transformer Extrapolation

Qin, Z., Zhong, Y., & Deng, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29855 · Cited by 2