A Survey of Multimodal Learning: Methods, Applications, and Future
The DOI matches its Crossref record; 122 of the 122 references it cites were found in a registry or an index.
Abstract
The multimodal interplay of the five fundamental senses—Sight, Hearing, Smell, Taste, and Touch—provides humans with superior environmental perception and learning skills. Adapted from the human perceptual system, multimodal machine learning tries to incorporate different forms of input, such as image, audio, and text, and determine their fundamental connections through joint modeling. As one of the future development forms of artificial intelligence, it is necessary to summarize the progress of multimodal machine learning. In this article, we start with the form of a multimodal combination and provide a comprehensive survey of the emerging subject of multimodal machine learning, covering representative research approaches, the most recent advancements, and their applications. Specifically, this article analyzes the relationship between different modalities in detail and sorts out the key issues in multimodal research from the application scenarios. Besides, we thoroughly reviewed state-of-the-art methods and datasets covered in multimodal learning research. We then identify the substantial challenges and potential developing directions in this field. Finally, given the comprehensive nature of this survey, both modality-specific and task-specific researchers can benefit from this survey and advance the field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 1×
“In 2015, the emergence of deep learning-based image captioning method [115] and visual question answering ( VQA ) task [3] has greatly promoted the development of visual-text modalities.”
Proceedings European Conference on Computer Vision 2002
Heyden, A., Sparr, G., Nielsen, M., & Johansen, P. (2002) · Lund University Publications (Lund University) · Cited by 5,153
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Places: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042
Proceedings. 1985 IEEE International Conference on Robotics and Automation
(1985) · doi:10.1109/robot.1985 · Cited by 3,673
Research works printed but not linked: 56
- Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr Yana Hasson Karel Lenc Arthur Mensch Katherine Millican Malcolm Reynolds et al. 2022. Flamingo: A visual language model for few-shot learning. Advances in Neural Information Processing Systems 35 (2022) 23716–23736.
- Dawit Mureja Argaw, Seunghyun Yoon, Fabian Caba Heilbron, Hanieh Deilamsalehy, Trung Bui, Zhaowen Wang, Franck Dernoncourt, and Joon Son Chung. 2024. Scaling up video summarization pretraining with large language models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8332–8341.
- Taivanbat Badamdorj, Mrigank Rochan, Yang Wang, and Li Cheng. 2021. Joint visual and audio learning for video highlight detection. In Proceedings of the IEEEF International Conference on Computer Vision. 8127–8137.
- Maxime Burchi and Radu Timofte. 2023. Audio-visual efficient conformer for robust speech recognition. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 2258–2267.
- Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2022. Zero-shot audio source separation through query-based learning from weakly-labeled data. In Proceedings of the Association for the Advance of Artificial Intelligence. 4441–4449.
- Angela Dai, Christian Diller, and Matthias Nießner. 2020. SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 849–858.
- Danny Driess Fei Xia Mehdi S. M. Sajjadi Corey Lynch Aakanksha Chowdhery Brian Ichter Ayzaan Wahid Jonathan Tompson Quan Vuong Tianhe Yu Wenlong Huang Yevgen Chebotar Pierre Sermanet Daniel Duckworth Sergey Levine Vincent Vanhoucke Karol Hausman Marc Toussaint Klaus Greff Andy Zeng Igor Mordatch and Pete Florence. 2023. PaLM-E: An embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning. 8469–8488.
- Ruihan Gao, Wenzhen Yuan, and Jun-Yan Zhu. 2023. Controllable visual-tactile synthesis. In Proceedings of the IEEE International Conference on Computer Vision. 7040–7052.
- Mallika Garg, Debashis Ghosh, and Pyari Mohan Pradhan. 2024. GestFormer: Multiscale wavelet pooling transformer network for dynamic hand gesture recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2473–2483.
- Rishabh Garg Ruohan Gao and Kristen Grauman. 2021. Geometry-aware multi-task learning for binaural audio generation from video. In Proceedings of the 32nd British Machine Vision Conference. 1.
- Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. ImageBind: One embedding space to bind them all. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 15180–15190.
- Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. 2023. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10867–10877.
- D. Hu, D. Wang, X. Li, F. Nie, and Q. Wang. 2019. Listen to the image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7964–7973.
- Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson W. H. Lau, Wanli Ouyang, and Wangmeng Zuo. 2023. CLIP2Point: Transfer CLIP to point cloud classification with image-depth pre-training. In Proceedings of the IEEE International Conference on Computer Vision. 22157–22167.
- Muhammad Rafiqul Islam, Manoranjan Paul, Michael Antolovich, and Ashad Kabir. 2019. Sports highlights generation using decomposed audio information. In IEEE International Conference on Multimedia & Expo Workshops. 579–584.
- Vahid Kiani and Hamid Reza Pourreza. 2013. Flexible soccer video summarization in compressed domain. In Proceedings of the IEEE International Conference on Computer and Knowledge Engineering. 213–218.
- Jing Yu Koh Daniel Fried and Russ R. Salakhutdinov. 2023. Generating images with multimodal language models. Advances in Neural Information Processing Systems 36 (2023) 21487–21506.
- Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. 2022. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 19108–19118.
- Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning. 12888–12900.
- Yunan Li, Huizhou Chen, Guanwen Feng, and Qiguang Miao. 2023. Learning robust representations with information bottleneck and memory network for RGB-D-based gesture recognition. In Proceedings of the IEEE International Conference on Computer Vision. 20968–20978.
- Y. Li, J. Zhu, R. Tedrake, and A. Torralba. 2019. Connecting touch and vision via cross-modal prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10601–10610.
- Zhaojian Li, Bin Zhao, and Yuan Yuan. 2024. Cyclic learning for binaural audio generation and localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 26669–26678.
- Miao Liu, Jing Wang, Xinyuan Qian, and Xiang Xie. 2024. Visually guided binaural audio generation with cross-modal consistency. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. 7980–7984.
- Yujie Lu Xianjun Yang Xiujun Li Xin Eric Wang and William Yang Wang. 2023. LLMScore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems 36 (2023) 23075–23093.
- Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2021. Thinking fast and slow: Efficient text-to-visual retrieval with transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9826–9836.
- Medhini Narasimhan Anna Rohrbach and Trevor Darrell. 2021. Clip-It! Language-guided video summarization. Advances in Neural Information Processing Systems 34 (2021) 13988–14000.
- Anshul Paigwar, David Sierra-Gonzalez, Özgür Erkent, and Christian Laugier. 2021. Frustum-pointpillars: A multi-stage approach for 3D object detection using rgb camera and lidar. In Proceedings of the IEEE International Conference on Computer Vision. 2926–2933.
- Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. 2022. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8238–8247.
- J. Perez-Rua, V. Vielzeuf, S. Pateux, M. Baccouche, and F. Jurie. 2019. MFAS: Multimodal fusion architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6959–6968.
- Jens Piekenbrinck, Alexander Hermans, Narunas Vaskevicius, Timm Linder, and Bastian Leibe. 2024. RGB-D cube R-CNN: 3D object detection with selective modality dropout. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1997–2006.
- Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. In Proceedings of International Conference on Machine Learning. 8821–8831.
- Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016. Generative adversarial text to image synthesis. In Proceedings of the International Conference on Machine Learning. 1060–1069.
- Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. 2024. TimeChat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 14313–14323.
- Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Besmira Nushi, and Ece Kamar. 2020. SQuINTing at VQA Models: Introspecting VQA models with sub-questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10003–10011.
- Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 14974–14983.
- M. Subedar, R. Krishnan, P. L. Meyer, O. Tickoo, and J. Huang. 2019. Uncertainty-aware audiovisual activity recognition using deep bayesian variational inference. In Proceedings of the IEEE International Conference on Computer Vision. 6300–6309.
- Weixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu, Yiran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. 2023. Learning audio-visual source localization via false negative aware contrastive learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6420–6429.
- Pichao Wang, Wanqing Li, Jun Wan, Philip Ogunbona, and Xinwang Liu. 2018. Cooperative training of deep aggregation networks for RGB-D action recognition. In Proceedings of the Association for the Advance of Artificial Intelligence. 7404–7411.
- S. Wang, J. Wu, X. Sun, W. Yuan, W. T. Freeman, J. B. Tenenbaum, and E. H. Adelson. 2018. 3D shape perception from monocular vision, touch, and shape priors. In Proceedings of the IEEE International Conference on Intelligent Robots and Systems. 1606–1613.
- W. Wang, D. Tran, and M. Feiszli. 2020. What makes training multi-modal classification networks hard?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12692–12702.
- Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and Liangwei Wang. 2020. On the general value of evidence, and bilingual scene-text visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10126–10135.
- Zhengwei Wang, Qi She, Tejo Chalasani, and Aljosa Smolic. 2020. CatNet: Class incremental 3D ConvNets for lifelong egocentric gesture recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 935–944.
- Y. Wu, L. Zhu, Y. Yan, and Y. Yang. 2019. Dual attention matching for audio-visual event localization. In Proceedings of the IEEE International Conference on Computer Vision. 6291–6299.
- Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. 2024. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7151–7161.
- Wenqiang Xu, Zhenjun Yu, Han Xue, Ruolin Ye, Siqiong Yao, and Cewu Lu. 2023. Visual-tactile sensing for in-hand object reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8803–8812.
- Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023. Vid2Seq: Large-scale pretraining of a visual language model for dense video captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10714–10726.
- Erkun Yang, Cheng Deng, Wei Liu, Xianglong Liu, Dacheng Tao, and Xinbo Gao. 2017. Pairwise relationship guided deep hashing for cross-modal retrieval. In Proceedings of the Association for the Advance of Artificial Intelligence. 1618–1625.
- Fengyu Yang, Jiacheng Zhang, and Andrew Owens. 2023. Generating visual scenes from touch. In Proceedings of the IEEE International Conference on Computer Vision. 22070–22080.
- K. Yang, B. Russell, and J. Salamon. 2020. Telling left from right: Learning spatial correspondence of sight and sound. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9929–9938.
- Qinghao Ye, Xiyue Shen, Yuan Gao, Zirui Wang, Qi Bi, Ping Li, and Guang Yang. 2021. Temporal cue guided video highlight detection with low-rank audio-visual fusion. In Proceedings of the IEEE International Conference on Computer Vision. 7950–7959.
- M. Zhang, W. Ren, Y. Piao, Z. Rong, and H. Lu. 2020. Select, supplement and focus for RGB-D saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3469–3478.
- H. Zhao, C. Gan, W. Ma, and A. Torralba. 2019. The sound of motions. In Proceedings of the IEEE International Conference on Computer Vision. 1735–1744.
- Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh H. McDermott, and Antonio Torralba. 2018. The sound of pixels. In Proceedings of the European Conference on Computer Vision. 587–604.
- Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. 2020. Sep-Stereo: Visually guided stereophonic audio generation by associating source separation. In Proceedings of the European Conference on Computer Vision. 52–69.
- Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, and Hanwang Zhang. 2020. More grounded image captioning by distilling image-text matching model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4776–4785.
- Lingyu Zhu and Esa Rahtu. 2022. Visually guided sound source separation and localization using self-supervised motion representations. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 1289–1299.
Cited by
Aligning generalization between humans and machines
Ilievski, F., Hammer, B., van Harmelen, F., Paaßen, B., Saralajew, S., Schmid, U., Biehl, M. L., Bolognesi, M., Dong, X., Gashteovski, K., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01109-4 · Cited by 58
Sun, J., Khalid, K. A. T., & Kay, C. S. (2025) · Future Technology · doi:10.55670/fpll.futech.4.3.12 · Cited by 26
He, Y., & Zhang, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-14164-z · Cited by 28
Combining Wearable Sensors and Artificial Intelligence in Cardiovascular Medicine
Maiorana, E., Campisi, P., Schena, E., & Massaroni, C. (2025) · IEEE Sensors Reviews · doi:10.1109/sr.2025.3623069 · Cited by 15
Large Language and Foundation Models for Machinery Health Monitoring: A Systematic Review
Tsallis, C., Papageorgas, P. G., Munteanu, R. A., & Dellagi, S. (2026) · Applied Sciences · doi:10.3390/app16052493 · Cited by 10
Ding, S., Xu, F., Jie, C., Liang, S., Zhang, H., & Huang, W. (2026) · Progress in Materials Science · doi:10.1016/j.pmatsci.2026.101733 · Cited by 12
Boosting crop disease recognition via automated image description generation and multimodal fusion
Xiong, S., Wang, L., Zhang, Y., Dong, P., Wang, B., Che, Y., Shi, L., & Si, H. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.111082 · Cited by 10
Guo, Q., Han, B., Chu, P., Wan, Y., & Zhang, J. (2025) · Agriculture · doi:10.3390/agriculture15151639 · Cited by 6
Cross-Lingual Adaptation for Vision-Language Model via Multimodal Semantic Distillation
Weng, Y., He, W., Dong, J., Chaomurilige, C., Liu, X., & Liu, Z. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3557678 · Cited by 15
Source code vulnerability detection based on deep learning: a review
Su, H., Xu, Z., Zhang, Y., & Tan, Q. (2026) · Cybersecurity · doi:10.1186/s42400-025-00518-7 · Cited by 12
Cross-modal consistency types in multimodal social data
Vaiani, L., Cagliero, L., Garza, P., & Ravagli, J. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113705 · Cited by 6
Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review
Khan, U., Nawaz, U., Qayyum, A., Baig, I., Alam, M., Ashraf, S. Q., Xie, Y., Khan, M. H., Bilal, M., & Qadir, J. (2026) · IEEE Access · doi:10.1109/access.2026.3729436 · Cited by 33
Safe multi-view graph convolutional network for semi-supervised classification
Xiang, D., Zhong, X., Sun, H., Tan, B., Gao, P., Zhang, J., Ling, J., Du, H., & Liang, N. (2026) · Neurocomputing · doi:10.1016/j.neucom.2025.132570 · Cited by 4
Elgohary, O., & Zhu-Tien (2025) · arXiv · doi:10.48550/arxiv.2509.05898 · Cited by 4
Multimodal learning on RGB-D image for precise litchi phenotyping and weight estimation
Yang, M., Chen, R., Chen, D., Wang, H., Wang, X., & Hu, F. (2025) · Plant Methods · doi:10.1186/s13007-025-01472-8 · Cited by 3
Mou, L., Zhen, S., Mao, S., & Ma, N. (2025) · doi:10.1145/3746277.3760407 · Cited by 4
Ghorbian, M., & Ghobaei‐Arani, M. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01304-5 · Cited by 6
Xu, Y., Song, J., Zhang, Z., Feng, L., Yang, Y., Liang, Y., & Guo, Y. (2026) · Sensors · doi:10.3390/s26061828 · Cited by 2
Araújo, H. C., Lisboa, M. O., Coura, P., Freitas, I. D. C., Villela, M. L. B., & Ferreira, R. (2026) · Revista Brasileira de Informática na Educação · doi:10.5753/rbie.2026.6311 · Cited by 2
Jiang, G.-F., Zhou, N., Wang, S.-M., & Ni, Y. (2026) · Results in Engineering · doi:10.1016/j.rineng.2026.111903 · Cited by 2
Double-incomplete multi-view clustering with self-induced semantic label diffusion
Chen, Z., Liu, M., Zhenjiao, L., Hong-wei, Z., Zhenjiao, L., Hong-wei, Z., & Zhao, L. (2026) · Information Sciences · doi:10.1016/j.ins.2026.123626 · Cited by 1
Multi-Scale Temporal Graph Contrastive Embedding for urban region representation
Li, Y., Niu, X., Zhu, J., Wen, S., & Min, F. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103464 · Cited by 3
Discrepancy-Aware Attention Network for Enhanced Audio-Visual Generalized Zero-Shot Learning
Yu, R., Gong, Y., Li, W., Sun, A., & Zheng, M. C. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3754966 · Cited by 2
Li, W., Wang, L., Zhao, M., Lü, X., & Yi, L. (2026) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2025.102694 · Cited by 4
A multimodal deep learning framework for nutritional estimation and health-oriented recipe analysis
Morales-Garzón, A., Quiñones-Muñoz, A., Gutiérrez‐Batista, K., & Martín-Bautista, M. J. (2026) · Multimedia Systems · doi:10.1007/s00530-025-02151-3 · Cited by 2
Modality-Aware Multimodal Fusion for Sleep Event Classification
Hayes, K., & Sun, Y. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401470 · Cited by 1
Theodorakopoulos, L., & Theodoropoulou, A. (2026) · Electronics · doi:10.3390/electronics15071405 · Cited by 1
Jia, C., Yang, L. S., Jin, W., Zhao, J., Zhang, C., & Li, Y. (2026) · Applied Energy · doi:10.1016/j.apenergy.2026.127772 · Cited by 1
Jolly, A., & Kumar, S. (2026) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2026.111203 · Cited by 2
Guest Editorial: Multimodal Representation and Reasoning for Social Computing
Feng, M., Cambria, E., Lin, Q., Shi, K., & Li, W. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3606567 · Cited by 1
Nadeem, A., Khan, M. Z., Sirshar, M., & Thirunavukkarasu, U. (2026) · Diagnostics · doi:10.3390/diagnostics16152377
Detection Of Malware Families and classification using Machine Learning
Pasha, C. A. A., Rahman, A., Abhishek, B. J., Kaushik, M., John, J., & Gopalakrishna (2025) · International Conference on Knowledge Engineering and Communication Systems (ICKECS) · doi:10.1109/ickecs65700.2025.11035552
Motif: A Text-Aware and Disentangled Mamba-Based Architecture for Multimodal Emotion RecognitionPreprint
Hu, B., & Li, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5351685
Zhao, B., Yang, Z., & Wu, Y. (2025) · International Conference on Mechatronics and Computer Technology Engineering (MCTE) · doi:10.1109/mcte67374.2025.11213654
Self-citationMulti-view information fusion based graph collaborative filtering recommendation
Mu, R., Li, X., & Zhang, J. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00408-5 · Cited by 3
Lin, Y., & Wang, X. (2025) · International Conference on Artificial Intelligence, Virtual Reality and Interaction Design · doi:10.1145/3777730.3777803
Improving Multimodal Sentiment Analysis via Stable Learning and Context Anchor Attention
Liu, Y., Li, M., Shi, J., & Kuang, Q. (2025) · International Conference on Advanced Computing and Intelligent Robotics Applications (ACIRA) · doi:10.1109/acira67680.2025.11334784
李, 志. (2026) · Computer Science and Application · doi:10.12677/csa.2026.161023
Yu, Z., Zhang, S., Liu, Y., & Chen, Q. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3661359
Immersive Language Learning: Exploring the Role of AR and VR in Education
Rakhimova, L. S., Saidjanova, Z., Khakkulov, Y. A., & Ahmad, M. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-3366-3.ch008
Multi-modal contrastive learning based on molecular and textual data for drug response prediction
Duan, M., Li, X., Hou, X., Qu, Y., Cui, H. N., & Zhang, Y. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108882
Multimodal sentiment analysis: emerging innovations, core challenges, and future directions
Dhotay, M., Dharrao, M., Deokate, S. T., Bongale, A. M., & Dharrao, D. S. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01141-2 · Cited by 2
AMTFusion: Boosting 3D Object Detection by Adaptive Multi-Modal Temporal Fusion and Augmentation
Tian, W., Du, Z., Zhao, X., & Yu, Q. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2026.3679483
Face off: A survey on multimodal fusion for secure and realistic facial de-identification
Zhu, H., Jain, D. K., Zhao, X., Li, M., Gan, C., & Zhang, H. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104343
Artificial Intelligence for Multimodal ESG Report Analysis: Detecting Corporate GreenwashingPreprint
Feng, W., Siau, K. L., & Chen, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6561557
Sun, L., & Zhou, Z. (2026) · International Journal of Humanoid Robotics · doi:10.1142/s0219843626400153
Human-centered perception methods: a survey on sensors, mechanisms and applications
Huang, J., Li, N., Yu, Z., Liang, B., Ye, D., & Zhao, X. (2026) · Measurement · doi:10.1016/j.measurement.2026.121694
Multimodal AI: PaLM-E’s Role in Vision–Language–Robotics & the Future of Efficient Fine-Tuning
Akhtar, Z. B. (2026) · Journal of Artificial Intelligence & Control Systems · doi:10.66967/jaics.2026.v2i103
Nguyen, P.-A., Le, T.-S., Le, D.-T., & Nguyen, C.-V. T. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12160-6 · Cited by 1
Ghayoumi, M., Ghazinour, K., Marrero, A., Barmas, D., Cook, C., May, M., Liu, C., Johnson, B., & Fofana, A. (2026) · Electronics · doi:10.3390/electronics15112421 · Cited by 2
Yang, B., Guo, C., Hao, K., Zhao, J., He, Z., & Tian, D. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3701293
MD-SRA: Multimodal Detection of SEO-based Redirection Attacks
Hur, M. G., Shin, J., & Min, M. (2026) · ACM Symposium on Applied Computing · doi:10.1145/3748522.3779877
Shen, Q., Xin, J., Dai, B. T., Yao, Z., Liu, X., Huang, Z., & Wang, Z. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3702501
CoTeach-CLIP: Cross-modal collaborative teachers for zero-shot point cloud recognition
Zuo, J., Zhang, H., Zhou, Q., Zhao, Y., Wang, H., & Ren, M. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134265
Evidential Uncertainty Modulated Adaptive Predictive Contrastive Learning for Multimodal Fusion
Mei, Q., Yu, H. Z., Yu, S., & Yang, Y. X. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810771
Wang, H., Wang, L., Zhou, S., Qin, Z., Hua, G., & Tang, W. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114282
Toward adaptive and high‑precision integrated pest management in the big data era
Yamanaka, T., Sun, J., & Kishi, S. (2026) · Current Opinion in Insect Science · doi:10.1016/j.cois.2026.101570
Audio-Related Multimodal Learning
Chen, W., Sun, Y., & Xu, K. (2026) · Springer eBooks · doi:10.1007/978-981-95-0836-5_21
Artificial Intelligence for Multimodal ESG Report Analysis: Detecting Corporate GreenwashingPreprint
Feng, W., Siau, K. L., & Chen, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6996371
DR²-Net: Adversarial Decoupling and Dynamic Reassembly forOOD-Robust RGB-IR FusionPreprint
Liang, G., Pang, K., Li, X., & Guo, X. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7036102
Jayaweera, V. L., Maduranga, M. W. P., Nayyar, A., & Saengudomlert, P. (2026) · Optical Switching and Networking · doi:10.1016/j.osn.2026.100876
SPSTLLM: A Statistical Prior-Enhanced Spatio-Temporal Large Language Model for traffic prediction
Wang, Y., Lv, Q., Li, Z., Li, J., Lv, Z., Liu, J., & Sun, D. (2026) · Advanced Engineering Informatics · doi:10.1016/j.aei.2026.105037
Self-citationMultimodal Representation Learning for Ionic Transport Barrier Prediction in Inorganic Compounds
Yuan, H., Yang, Z., Zhang, W., Ma, C., Shi, S., & Liu, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3384-7_31
UMSS: Towards Unsupervised Multi-modal Semantic Segmentation
Zhang, H., Nguyen, T. D., Wang, X., Liu, M., & Wang, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37174-4_30
Zubair, M., Zheng, H., Armstrong, G. W., Shen, L. Q., Wilson, G. M., Tian, Y., & Xingquan Zhu, A. (2026) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2026.3713843
Multimodal information fusion for food nutrition estimation: A survey
Teng, X., Min, W., Liu, C., & Li, H. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104621 · Cited by 1
Ye, Z., Peng, Z., Gowda, S. N., Yan, Y., Xu, H., & Shao, L. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3709493
Multimodal Data Structure Based on Dimensions of Mathematical Learning
Téllez, N. R., Herrera, V. F., & Pérez-Domínguez, L. A. (2026) · Studies in big data · doi:10.1007/978-3-032-31204-4_14
Hung, V. T., Thuyen, P. T. L., & Khoa, N. D. (2026) · International Conference on Applying New Technology in Green Buildings (ATiGB) · doi:10.1109/atigb70203.2026.11628430
Yang, L., Zhang, H., Zhang, Y., Wu, B., Wang, J., Tang, J., & Zhang, H. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.116228
Wu, P., Liu, J., Lin, Z., Wang, Z., Chu, C., & Yi, Z. (2026) · Lecture notes in mechanical engineering · doi:10.1007/978-981-92-3086-0_15
CNN-LSTM detection and short-term prediction of concrete crack severity
Tang, J., Guo, G., Wang, T., Wang, Q., Meng, Q., & Ni, S. (2026) · Proceedings of the Institution of Civil Engineers - Construction Materials · doi:10.1680/jcoma.25.00109
Lai, C., Zhang, H., Cheng, C., Peng, J., & Yuan, X. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag448
Dong, S.-H., Chen, J.-C., Feng, W.-J., Xu, R., & Wei, B. (2026) · Fall Joint Computer Conference · doi:10.1109/jcc72984.2026.00016
Liu, Y., Zhang, C., Lin, Z., Ning, C., Liang, X., & Gui, W. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109552
Multimodal Knowledge Graph Link Prediction Model Based on Sequential Fusion
Zhao, Y., & Liang, Y. (2026) · 電腦學刊 · doi:10.63367/199115992026083704002
Deep Learning-Based Multi-Source Data Fusion Methods for Intelligent Information Processing
Lin, G. (2026) · International Conference on Artificial Intelligence, Virtual Reality and Interaction Design · doi:10.1145/3820915.3820932
Geometry-Guided Cross-Modal Consistency Fusion for Unsupervised Industrial Surface Anomaly Detection
An, Y., Man, G., Zheng, Q., & Liu, L. (2026) · IEEE Access · doi:10.1109/access.2026.3735547
ViSA: Subquadratic cross-modal attention with dual-path subtractive fusion
Sun, H., Jiang, M., & Lu, M. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.117097
$$\text {G}^3$$AFT: Glance Guided Gradient Aligned Fine-Tuning for Visual Autoregressive Models
Zhang, J., Guo, X., Ma, X., & Huang, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37520-9_6
Hadi, A. A., & Supreethi, K. P. (2026) · Discover Informatics · doi:10.1007/s44564-026-00022-1
DatasetOS: An Agentic Framework For Multimodal Dataset Construction
Alsreidi, B., Aldoboni, G., Nassar, L., & Karray, F. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831193
A survey of LLM reasoning in healthcare and medicine: Individual modeling and collaborative agents
Lin, Q., Xiao, L., Pu, B., Shi, K., Du, Y., Xu, J., He, K., Zhao, S., Zhang, J., Cambria, E., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104827 · Cited by 7
Chen, D., Liu, Z.-T., Luo, L., & She, J. (2026) · Asian Control Conference (ASCC) · doi:10.1109/ascc70886.2026.11702377
Multi-modal adaptive weighted distillation for industrial equipment health monitoring
Yu, H., Yan, W., Liu, L., Wan, Q., Liu, W., Zhou, W., Zhang, Z., Hu, Y., & Weifang, C. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.116436
EAM-DG: emotion-aware memory network with dual-gating mechanism for dialogue emotion recognition
Wu, T., Li, Y., Zheng, Y., Wang, K., Ma, Y., Li, D., & Li, S. (2026) · Multimedia Systems · doi:10.1007/s00530-026-02688-x
Abolfazlia, Z., & Abutalebi, H. R. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.116676
Lai, C., Zhang, H., Deng, D., Peng, J., & Yuan, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37679-4_39
Do, H., Chun, C., & Kim, T. (2026) · Digital · doi:10.3390/digital6040087
Özgür, T., & Atabeyoğlu, Ö. (2026) · Özgür Yayınları eBooks · doi:10.58830/ozgur.pub1564.c6396
Ji, Y., & Yang, S. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.116774
Metadata
- Authors
- Yuan, Y., Li, Z., & Zhao, B.
- Affiliations
- Northwestern Polytechnical University (CN)
- Venue
- ACM Computing Surveys
- Published
- 2025-02-20
- Volume
- 57
- Issue
- 7
- Pages
- 1-34
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4406552230
- Semantic Scholar
- 1f31434605e5db7f8033149ab0b713fc6829d67d
- Cited by
- 90
- References
- 82
- Citations by year
- 2026: 69 · 2025: 21
- FWCI
- 31.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Computing Surveys (publishedVersion)
- Funders
- National Science Fund for Distinguished Young Scholars (61825603)
- National Natural Science Foundation of China (62106183)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, cross-modal relationships, multimodal dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.