CoChat

A Survey of Multimodal Learning: Methods, Applications, and Future

Yuan, Yuan, Li, Zhaojian, Zhao, Bin · 2025 · ACM Computing Surveys

The DOI matches its Crossref record; 121 of the 121 references it cites were found in a registry or an index.

Abstract

The multimodal interplay of the five fundamental senses—Sight, Hearing, Smell, Taste, and Touch—provides humans with superior environmental perception and learning skills. Adapted from the human perceptual system, multimodal machine learning tries to incorporate different forms of input, such as image, audio, and text, and determine their fundamental connections through joint modeling. As one of the future development forms of artificial intelligence, it is necessary to summarize the progress of multimodal machine learning. In this article, we start with the form of a multimodal combination and provide a comprehensive survey of the emerging subject of multimodal machine learning, covering representative research approaches, the most recent advancements, and their applications. Specifically, this article analyzes the relationship between different modalities in detail and sorts out the key issues in multimodal research from the application scenarios. Besides, we thoroughly reviewed state-of-the-art methods and datasets covered in multimodal learning research. We then identify the substantial challenges and potential developing directions in this field. Finally, given the comprehensive nature of this survey, both modality-specific and task-specific researchers can benefit from this survey and advance the field.

Overview

Integrity signals

Registry

Crossref · 10.1145/3713070

Cites

121 of 121 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Research works printed but not linked: 56

  1. Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr Yana Hasson Karel Lenc Arthur Mensch Katherine Millican Malcolm Reynolds et al. 2022. Flamingo: A visual language model for few-shot learning. Advances in Neural Information Processing Systems 35 (2022) 23716–23736.
  2. Dawit Mureja Argaw, Seunghyun Yoon, Fabian Caba Heilbron, Hanieh Deilamsalehy, Trung Bui, Zhaowen Wang, Franck Dernoncourt, and Joon Son Chung. 2024. Scaling up video summarization pretraining with large language models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8332–8341.
  3. Taivanbat Badamdorj, Mrigank Rochan, Yang Wang, and Li Cheng. 2021. Joint visual and audio learning for video highlight detection. In Proceedings of the IEEEF International Conference on Computer Vision. 8127–8137.
  4. Maxime Burchi and Radu Timofte. 2023. Audio-visual efficient conformer for robust speech recognition. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 2258–2267.
  5. Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2022. Zero-shot audio source separation through query-based learning from weakly-labeled data. In Proceedings of the Association for the Advance of Artificial Intelligence. 4441–4449.
  6. Angela Dai, Christian Diller, and Matthias Nießner. 2020. SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 849–858.
  7. Danny Driess Fei Xia Mehdi S. M. Sajjadi Corey Lynch Aakanksha Chowdhery Brian Ichter Ayzaan Wahid Jonathan Tompson Quan Vuong Tianhe Yu Wenlong Huang Yevgen Chebotar Pierre Sermanet Daniel Duckworth Sergey Levine Vincent Vanhoucke Karol Hausman Marc Toussaint Klaus Greff Andy Zeng Igor Mordatch and Pete Florence. 2023. PaLM-E: An embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning. 8469–8488.
  8. Ruihan Gao, Wenzhen Yuan, and Jun-Yan Zhu. 2023. Controllable visual-tactile synthesis. In Proceedings of the IEEE International Conference on Computer Vision. 7040–7052.
  9. Mallika Garg, Debashis Ghosh, and Pyari Mohan Pradhan. 2024. GestFormer: Multiscale wavelet pooling transformer network for dynamic hand gesture recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2473–2483.
  10. Rishabh Garg Ruohan Gao and Kristen Grauman. 2021. Geometry-aware multi-task learning for binaural audio generation from video. In Proceedings of the 32nd British Machine Vision Conference. 1.
  11. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. ImageBind: One embedding space to bind them all. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 15180–15190.
  12. Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. 2023. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10867–10877.
  13. D. Hu, D. Wang, X. Li, F. Nie, and Q. Wang. 2019. Listen to the image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7964–7973.
  14. Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson W. H. Lau, Wanli Ouyang, and Wangmeng Zuo. 2023. CLIP2Point: Transfer CLIP to point cloud classification with image-depth pre-training. In Proceedings of the IEEE International Conference on Computer Vision. 22157–22167.
  15. Muhammad Rafiqul Islam, Manoranjan Paul, Michael Antolovich, and Ashad Kabir. 2019. Sports highlights generation using decomposed audio information. In IEEE International Conference on Multimedia & Expo Workshops. 579–584.
  16. Vahid Kiani and Hamid Reza Pourreza. 2013. Flexible soccer video summarization in compressed domain. In Proceedings of the IEEE International Conference on Computer and Knowledge Engineering. 213–218.
  17. Jing Yu Koh Daniel Fried and Russ R. Salakhutdinov. 2023. Generating images with multimodal language models. Advances in Neural Information Processing Systems 36 (2023) 21487–21506.
  18. Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. 2022. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 19108–19118.
  19. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning. 12888–12900.
  20. Yunan Li, Huizhou Chen, Guanwen Feng, and Qiguang Miao. 2023. Learning robust representations with information bottleneck and memory network for RGB-D-based gesture recognition. In Proceedings of the IEEE International Conference on Computer Vision. 20968–20978.
  21. Y. Li, J. Zhu, R. Tedrake, and A. Torralba. 2019. Connecting touch and vision via cross-modal prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10601–10610.
  22. Zhaojian Li, Bin Zhao, and Yuan Yuan. 2024. Cyclic learning for binaural audio generation and localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 26669–26678.
  23. Miao Liu, Jing Wang, Xinyuan Qian, and Xiang Xie. 2024. Visually guided binaural audio generation with cross-modal consistency. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. 7980–7984.
  24. Yujie Lu Xianjun Yang Xiujun Li Xin Eric Wang and William Yang Wang. 2023. LLMScore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems 36 (2023) 23075–23093.
  25. Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2021. Thinking fast and slow: Efficient text-to-visual retrieval with transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9826–9836.
  26. Medhini Narasimhan Anna Rohrbach and Trevor Darrell. 2021. Clip-It! Language-guided video summarization. Advances in Neural Information Processing Systems 34 (2021) 13988–14000.
  27. Anshul Paigwar, David Sierra-Gonzalez, Özgür Erkent, and Christian Laugier. 2021. Frustum-pointpillars: A multi-stage approach for 3D object detection using rgb camera and lidar. In Proceedings of the IEEE International Conference on Computer Vision. 2926–2933.
  28. Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. 2022. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8238–8247.
  29. J. Perez-Rua, V. Vielzeuf, S. Pateux, M. Baccouche, and F. Jurie. 2019. MFAS: Multimodal fusion architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6959–6968.
  30. Jens Piekenbrinck, Alexander Hermans, Narunas Vaskevicius, Timm Linder, and Bastian Leibe. 2024. RGB-D cube R-CNN: 3D object detection with selective modality dropout. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1997–2006.
  31. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. In Proceedings of International Conference on Machine Learning. 8821–8831.
  32. Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016. Generative adversarial text to image synthesis. In Proceedings of the International Conference on Machine Learning. 1060–1069.
  33. Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. 2024. TimeChat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 14313–14323.
  34. Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Besmira Nushi, and Ece Kamar. 2020. SQuINTing at VQA Models: Introspecting VQA models with sub-questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10003–10011.
  35. Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 14974–14983.
  36. M. Subedar, R. Krishnan, P. L. Meyer, O. Tickoo, and J. Huang. 2019. Uncertainty-aware audiovisual activity recognition using deep bayesian variational inference. In Proceedings of the IEEE International Conference on Computer Vision. 6300–6309.
  37. Weixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu, Yiran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. 2023. Learning audio-visual source localization via false negative aware contrastive learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6420–6429.
  38. Pichao Wang, Wanqing Li, Jun Wan, Philip Ogunbona, and Xinwang Liu. 2018. Cooperative training of deep aggregation networks for RGB-D action recognition. In Proceedings of the Association for the Advance of Artificial Intelligence. 7404–7411.
  39. S. Wang, J. Wu, X. Sun, W. Yuan, W. T. Freeman, J. B. Tenenbaum, and E. H. Adelson. 2018. 3D shape perception from monocular vision, touch, and shape priors. In Proceedings of the IEEE International Conference on Intelligent Robots and Systems. 1606–1613.
  40. W. Wang, D. Tran, and M. Feiszli. 2020. What makes training multi-modal classification networks hard?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12692–12702.
  41. Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and Liangwei Wang. 2020. On the general value of evidence, and bilingual scene-text visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10126–10135.
  42. Zhengwei Wang, Qi She, Tejo Chalasani, and Aljosa Smolic. 2020. CatNet: Class incremental 3D ConvNets for lifelong egocentric gesture recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 935–944.
  43. Y. Wu, L. Zhu, Y. Yan, and Y. Yang. 2019. Dual attention matching for audio-visual event localization. In Proceedings of the IEEE International Conference on Computer Vision. 6291–6299.
  44. Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. 2024. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7151–7161.
  45. Wenqiang Xu, Zhenjun Yu, Han Xue, Ruolin Ye, Siqiong Yao, and Cewu Lu. 2023. Visual-tactile sensing for in-hand object reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8803–8812.
  46. Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023. Vid2Seq: Large-scale pretraining of a visual language model for dense video captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10714–10726.
  47. Erkun Yang, Cheng Deng, Wei Liu, Xianglong Liu, Dacheng Tao, and Xinbo Gao. 2017. Pairwise relationship guided deep hashing for cross-modal retrieval. In Proceedings of the Association for the Advance of Artificial Intelligence. 1618–1625.
  48. Fengyu Yang, Jiacheng Zhang, and Andrew Owens. 2023. Generating visual scenes from touch. In Proceedings of the IEEE International Conference on Computer Vision. 22070–22080.
  49. K. Yang, B. Russell, and J. Salamon. 2020. Telling left from right: Learning spatial correspondence of sight and sound. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9929–9938.
  50. Qinghao Ye, Xiyue Shen, Yuan Gao, Zirui Wang, Qi Bi, Ping Li, and Guang Yang. 2021. Temporal cue guided video highlight detection with low-rank audio-visual fusion. In Proceedings of the IEEE International Conference on Computer Vision. 7950–7959.
  51. M. Zhang, W. Ren, Y. Piao, Z. Rong, and H. Lu. 2020. Select, supplement and focus for RGB-D saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3469–3478.
  52. H. Zhao, C. Gan, W. Ma, and A. Torralba. 2019. The sound of motions. In Proceedings of the IEEE International Conference on Computer Vision. 1735–1744.
  53. Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh H. McDermott, and Antonio Torralba. 2018. The sound of pixels. In Proceedings of the European Conference on Computer Vision. 587–604.
  54. Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. 2020. Sep-Stereo: Visually guided stereophonic audio generation by associating source separation. In Proceedings of the European Conference on Computer Vision. 52–69.
  55. Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, and Hanwang Zhang. 2020. More grounded image captioning by distilling image-text matching model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4776–4785.
  56. Lingyu Zhu and Esa Rahtu. 2022. Visually guided sound source separation and localization using self-supervised motion representations. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 1289–1299.

Cited by

See all 91

Metadata

Authors
Yuan, Y., Li, Z., & Zhao, B.
Affiliations
Northwestern Polytechnical University (CN)
Venue
ACM Computing Surveys
Published
2025-02-20
Volume
57
Issue
7
Pages
1-34
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W4406552230
Cited by
90
References
82
Citations by year
2026: 69 · 2025: 21
FWCI
31.28
Type
journal article
Language
en
Open access
closed
Locations
ACM Computing Surveys (publishedVersion)
Funders
  • National Science Fund for Distinguished Young Scholars (61825603)
  • National Natural Science Foundation of China (62106183)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
multimodal learning, multimodal fusion, cross-modal relationships, multimodal dataset

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.