A Survey of Multimodal Learning: Methods, Applications, and Future
The DOI matches its Crossref record; 121 of the 121 references it cites were found in a registry or an index.
Abstract
The multimodal interplay of the five fundamental senses—Sight, Hearing, Smell, Taste, and Touch—provides humans with superior environmental perception and learning skills. Adapted from the human perceptual system, multimodal machine learning tries to incorporate different forms of input, such as image, audio, and text, and determine their fundamental connections through joint modeling. As one of the future development forms of artificial intelligence, it is necessary to summarize the progress of multimodal machine learning. In this article, we start with the form of a multimodal combination and provide a comprehensive survey of the emerging subject of multimodal machine learning, covering representative research approaches, the most recent advancements, and their applications. Specifically, this article analyzes the relationship between different modalities in detail and sorts out the key issues in multimodal research from the application scenarios. Besides, we thoroughly reviewed state-of-the-art methods and datasets covered in multimodal learning research. We then identify the substantial challenges and potential developing directions in this field. Finally, given the comprehensive nature of this survey, both modality-specific and task-specific researchers can benefit from this survey and advance the field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 1×
“In 2015, the emergence of deep learning-based image captioning method [115] and visual question answering ( VQA ) task [3] has greatly promoted the development of visual-text modalities.”
Proceedings European Conference on Computer Vision 2002
Heyden, A., Sparr, G., Nielsen, M., & Johansen, P. (2002) · Lund University Publications (Lund University) · Cited by 5,153
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Places: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042
Proceedings. 1985 IEEE International Conference on Robotics and Automation
(1985) · doi:10.1109/robot.1985 · Cited by 3,673
Research works printed but not linked: 56
- Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr Yana Hasson Karel Lenc Arthur Mensch Katherine Millican Malcolm Reynolds et al. 2022. Flamingo: A visual language model for few-shot learning. Advances in Neural Information Processing Systems 35 (2022) 23716–23736.
- Dawit Mureja Argaw, Seunghyun Yoon, Fabian Caba Heilbron, Hanieh Deilamsalehy, Trung Bui, Zhaowen Wang, Franck Dernoncourt, and Joon Son Chung. 2024. Scaling up video summarization pretraining with large language models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8332–8341.
- Taivanbat Badamdorj, Mrigank Rochan, Yang Wang, and Li Cheng. 2021. Joint visual and audio learning for video highlight detection. In Proceedings of the IEEEF International Conference on Computer Vision. 8127–8137.
- Maxime Burchi and Radu Timofte. 2023. Audio-visual efficient conformer for robust speech recognition. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 2258–2267.
- Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2022. Zero-shot audio source separation through query-based learning from weakly-labeled data. In Proceedings of the Association for the Advance of Artificial Intelligence. 4441–4449.
- Angela Dai, Christian Diller, and Matthias Nießner. 2020. SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 849–858.
- Danny Driess Fei Xia Mehdi S. M. Sajjadi Corey Lynch Aakanksha Chowdhery Brian Ichter Ayzaan Wahid Jonathan Tompson Quan Vuong Tianhe Yu Wenlong Huang Yevgen Chebotar Pierre Sermanet Daniel Duckworth Sergey Levine Vincent Vanhoucke Karol Hausman Marc Toussaint Klaus Greff Andy Zeng Igor Mordatch and Pete Florence. 2023. PaLM-E: An embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning. 8469–8488.
- Ruihan Gao, Wenzhen Yuan, and Jun-Yan Zhu. 2023. Controllable visual-tactile synthesis. In Proceedings of the IEEE International Conference on Computer Vision. 7040–7052.
- Mallika Garg, Debashis Ghosh, and Pyari Mohan Pradhan. 2024. GestFormer: Multiscale wavelet pooling transformer network for dynamic hand gesture recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2473–2483.
- Rishabh Garg Ruohan Gao and Kristen Grauman. 2021. Geometry-aware multi-task learning for binaural audio generation from video. In Proceedings of the 32nd British Machine Vision Conference. 1.
- Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. ImageBind: One embedding space to bind them all. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 15180–15190.
- Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. 2023. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10867–10877.
- D. Hu, D. Wang, X. Li, F. Nie, and Q. Wang. 2019. Listen to the image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7964–7973.
- Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson W. H. Lau, Wanli Ouyang, and Wangmeng Zuo. 2023. CLIP2Point: Transfer CLIP to point cloud classification with image-depth pre-training. In Proceedings of the IEEE International Conference on Computer Vision. 22157–22167.
- Muhammad Rafiqul Islam, Manoranjan Paul, Michael Antolovich, and Ashad Kabir. 2019. Sports highlights generation using decomposed audio information. In IEEE International Conference on Multimedia & Expo Workshops. 579–584.
- Vahid Kiani and Hamid Reza Pourreza. 2013. Flexible soccer video summarization in compressed domain. In Proceedings of the IEEE International Conference on Computer and Knowledge Engineering. 213–218.
- Jing Yu Koh Daniel Fried and Russ R. Salakhutdinov. 2023. Generating images with multimodal language models. Advances in Neural Information Processing Systems 36 (2023) 21487–21506.
- Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. 2022. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 19108–19118.
- Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning. 12888–12900.
- Yunan Li, Huizhou Chen, Guanwen Feng, and Qiguang Miao. 2023. Learning robust representations with information bottleneck and memory network for RGB-D-based gesture recognition. In Proceedings of the IEEE International Conference on Computer Vision. 20968–20978.
- Y. Li, J. Zhu, R. Tedrake, and A. Torralba. 2019. Connecting touch and vision via cross-modal prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10601–10610.
- Zhaojian Li, Bin Zhao, and Yuan Yuan. 2024. Cyclic learning for binaural audio generation and localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 26669–26678.
- Miao Liu, Jing Wang, Xinyuan Qian, and Xiang Xie. 2024. Visually guided binaural audio generation with cross-modal consistency. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. 7980–7984.
- Yujie Lu Xianjun Yang Xiujun Li Xin Eric Wang and William Yang Wang. 2023. LLMScore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems 36 (2023) 23075–23093.
- Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2021. Thinking fast and slow: Efficient text-to-visual retrieval with transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9826–9836.
- Medhini Narasimhan Anna Rohrbach and Trevor Darrell. 2021. Clip-It! Language-guided video summarization. Advances in Neural Information Processing Systems 34 (2021) 13988–14000.
- Anshul Paigwar, David Sierra-Gonzalez, Özgür Erkent, and Christian Laugier. 2021. Frustum-pointpillars: A multi-stage approach for 3D object detection using rgb camera and lidar. In Proceedings of the IEEE International Conference on Computer Vision. 2926–2933.
- Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. 2022. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8238–8247.
- J. Perez-Rua, V. Vielzeuf, S. Pateux, M. Baccouche, and F. Jurie. 2019. MFAS: Multimodal fusion architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6959–6968.
- Jens Piekenbrinck, Alexander Hermans, Narunas Vaskevicius, Timm Linder, and Bastian Leibe. 2024. RGB-D cube R-CNN: 3D object detection with selective modality dropout. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1997–2006.
- Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. In Proceedings of International Conference on Machine Learning. 8821–8831.
- Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016. Generative adversarial text to image synthesis. In Proceedings of the International Conference on Machine Learning. 1060–1069.
- Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. 2024. TimeChat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 14313–14323.
- Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Besmira Nushi, and Ece Kamar. 2020. SQuINTing at VQA Models: Introspecting VQA models with sub-questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10003–10011.
- Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 14974–14983.
- M. Subedar, R. Krishnan, P. L. Meyer, O. Tickoo, and J. Huang. 2019. Uncertainty-aware audiovisual activity recognition using deep bayesian variational inference. In Proceedings of the IEEE International Conference on Computer Vision. 6300–6309.
- Weixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu, Yiran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. 2023. Learning audio-visual source localization via false negative aware contrastive learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6420–6429.
- Pichao Wang, Wanqing Li, Jun Wan, Philip Ogunbona, and Xinwang Liu. 2018. Cooperative training of deep aggregation networks for RGB-D action recognition. In Proceedings of the Association for the Advance of Artificial Intelligence. 7404–7411.
- S. Wang, J. Wu, X. Sun, W. Yuan, W. T. Freeman, J. B. Tenenbaum, and E. H. Adelson. 2018. 3D shape perception from monocular vision, touch, and shape priors. In Proceedings of the IEEE International Conference on Intelligent Robots and Systems. 1606–1613.
- W. Wang, D. Tran, and M. Feiszli. 2020. What makes training multi-modal classification networks hard?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12692–12702.
- Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and Liangwei Wang. 2020. On the general value of evidence, and bilingual scene-text visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10126–10135.
- Zhengwei Wang, Qi She, Tejo Chalasani, and Aljosa Smolic. 2020. CatNet: Class incremental 3D ConvNets for lifelong egocentric gesture recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 935–944.
- Y. Wu, L. Zhu, Y. Yan, and Y. Yang. 2019. Dual attention matching for audio-visual event localization. In Proceedings of the IEEE International Conference on Computer Vision. 6291–6299.
- Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. 2024. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7151–7161.
- Wenqiang Xu, Zhenjun Yu, Han Xue, Ruolin Ye, Siqiong Yao, and Cewu Lu. 2023. Visual-tactile sensing for in-hand object reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8803–8812.
- Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023. Vid2Seq: Large-scale pretraining of a visual language model for dense video captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10714–10726.
- Erkun Yang, Cheng Deng, Wei Liu, Xianglong Liu, Dacheng Tao, and Xinbo Gao. 2017. Pairwise relationship guided deep hashing for cross-modal retrieval. In Proceedings of the Association for the Advance of Artificial Intelligence. 1618–1625.
- Fengyu Yang, Jiacheng Zhang, and Andrew Owens. 2023. Generating visual scenes from touch. In Proceedings of the IEEE International Conference on Computer Vision. 22070–22080.
- K. Yang, B. Russell, and J. Salamon. 2020. Telling left from right: Learning spatial correspondence of sight and sound. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9929–9938.
- Qinghao Ye, Xiyue Shen, Yuan Gao, Zirui Wang, Qi Bi, Ping Li, and Guang Yang. 2021. Temporal cue guided video highlight detection with low-rank audio-visual fusion. In Proceedings of the IEEE International Conference on Computer Vision. 7950–7959.
- M. Zhang, W. Ren, Y. Piao, Z. Rong, and H. Lu. 2020. Select, supplement and focus for RGB-D saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3469–3478.
- H. Zhao, C. Gan, W. Ma, and A. Torralba. 2019. The sound of motions. In Proceedings of the IEEE International Conference on Computer Vision. 1735–1744.
- Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh H. McDermott, and Antonio Torralba. 2018. The sound of pixels. In Proceedings of the European Conference on Computer Vision. 587–604.
- Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. 2020. Sep-Stereo: Visually guided stereophonic audio generation by associating source separation. In Proceedings of the European Conference on Computer Vision. 52–69.
- Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, and Hanwang Zhang. 2020. More grounded image captioning by distilling image-text matching model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4776–4785.
- Lingyu Zhu and Esa Rahtu. 2022. Visually guided sound source separation and localization using self-supervised motion representations. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 1289–1299.
Cited by
Aligning generalization between humans and machines
Ilievski, F., Hammer, B., van Harmelen, F., Paaßen, B., Saralajew, S., Schmid, U., Biehl, M. L., Bolognesi, M., Dong, X., Gashteovski, K., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01109-4 · Cited by 58
Sun, J., Khalid, K. A. T., & Kay, C. S. (2025) · Future Technology · doi:10.55670/fpll.futech.4.3.12 · Cited by 26
He, Y., & Zhang, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-14164-z · Cited by 28
Combining Wearable Sensors and Artificial Intelligence in Cardiovascular Medicine
Maiorana, E., Campisi, P., Schena, E., & Massaroni, C. (2025) · IEEE Sensors Reviews · doi:10.1109/sr.2025.3623069 · Cited by 15
Large Language and Foundation Models for Machinery Health Monitoring: A Systematic Review
Tsallis, C., Papageorgas, P. G., Munteanu, R. A., & Dellagi, S. (2026) · Applied Sciences · doi:10.3390/app16052493 · Cited by 10
Metadata
- Authors
- Yuan, Y., Li, Z., & Zhao, B.
- Affiliations
- Northwestern Polytechnical University (CN)
- Venue
- ACM Computing Surveys
- Published
- 2025-02-20
- Volume
- 57
- Issue
- 7
- Pages
- 1-34
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4406552230
- Semantic Scholar
- 1f31434605e5db7f8033149ab0b713fc6829d67d
- Cited by
- 90
- References
- 82
- Citations by year
- 2026: 69 · 2025: 21
- FWCI
- 31.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Computing Surveys (publishedVersion)
- Funders
- National Science Fund for Distinguished Young Scholars (61825603)
- National Natural Science Foundation of China (62106183)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, cross-modal relationships, multimodal dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Survey of Multimodal Learning: Methods, Applications, and Future. https://app.cochat.ai/papers/10.1145/3713070
- The paper
- Yuan, Y., Li, Z., & Zhao, B. (2025). A Survey of Multimodal Learning: Methods, Applications, and Future. ACM Computing Surveys. https://doi.org/10.1145/3713070
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.