A Survey of Multimodal Learning: Methods, Applications, and Future
The DOI matches its Crossref record; 121 of the 121 references it cites were found in a registry or an index.
Abstract
The multimodal interplay of the five fundamental senses—Sight, Hearing, Smell, Taste, and Touch—provides humans with superior environmental perception and learning skills. Adapted from the human perceptual system, multimodal machine learning tries to incorporate different forms of input, such as image, audio, and text, and determine their fundamental connections through joint modeling. As one of the future development forms of artificial intelligence, it is necessary to summarize the progress of multimodal machine learning. In this article, we start with the form of a multimodal combination and provide a comprehensive survey of the emerging subject of multimodal machine learning, covering representative research approaches, the most recent advancements, and their applications. Specifically, this article analyzes the relationship between different modalities in detail and sorts out the key issues in multimodal research from the application scenarios. Besides, we thoroughly reviewed state-of-the-art methods and datasets covered in multimodal learning research. We then identify the substantial challenges and potential developing directions in this field. Finally, given the comprehensive nature of this survey, both modality-specific and task-specific researchers can benefit from this survey and advance the field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 1×
“In 2015, the emergence of deep learning-based image captioning method [115] and visual question answering ( VQA ) task [3] has greatly promoted the development of visual-text modalities.”
Proceedings European Conference on Computer Vision 2002
Heyden, A., Sparr, G., Nielsen, M., & Johansen, P. (2002) · Lund University Publications (Lund University) · Cited by 5,153
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Places: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042
Proceedings. 1985 IEEE International Conference on Robotics and Automation
(1985) · doi:10.1109/robot.1985 · Cited by 3,673
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021 · cited 2×
“To solve the above problems, Zhang et al. [150] proposed a StackGAN to generate images in two independent stages.”
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511
Microsoft Kinect Sensor and Its Effect
Zhang, Z. (2012) · IEEE Multimedia · doi:10.1109/mmul.2012.24 · Cited by 2,487
Livescu, K., Bezman, A., Borges, N., Yung, L.-Y., Çetin, Ö., Frankel, J., King, S., Magimai.-Doss, M., Chi, X., & Lavoie, L. (2007) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 2,145
Proceedings IEEE International Conference on Acoustics Speech and Signal Processing
Zheng, F., McLaughlin, S., & Mulgrew, B. (1991) · Cited by 1,905
Proceedings of the IEEE International Conference on Image Processing
Calway, A. D. (1996) · The Institute of Electrical and Electronics Engineers · Cited by 1,578
Every Picture Tells a Story: Generating Sentences from Images
Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., & Forsyth, D. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_2 · Cited by 1,204
Deep Modular Co-Attention Networks for Visual Question Answering
Yu, Z., Yu, J., Cui, Y., Tao, D., & Tian, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00644 · Cited by 964
Arandjelovic, R., & Zisserman, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 782
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Owens, A., & Efros, A. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_39 · Cited by 704
MirrorGAN: Learning Text-To-Image Generation by Redescription
Qiao, T., Zhang, J., Xu, D., & Tao, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00160 · Cited by 644 · cited 2×
“To end this, Qiao et al. [92] proposed a text-to-image-to-text framework MirrorGAN that can keep the semantics of image text consistent.”
Proceedings of the 31st ACM International Conference on Multimedia
(2023) · doi:10.1145/3581783 · Cited by 588
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
Zhang, H., Li, X., & Bing, L. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2023.emnlp-demo.49 · Cited by 586
Deep Supervised Cross-Modal Retrieval
Zhen, L., Hu, P., Wang, X., & Peng, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01064 · Cited by 478 · cited 1×
“Zhen et al. [159] proposed a new cross-modal retrieval method (DSCMR) in the cross-modal retrieval task to better learn the characteristics of modal invariance, which can find a common representation space for the modal.”
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469
UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders
Zhang, J., Fan, D.-P., Dai, Y., Anwar, S., Saleh, F. S., Zhang, T., & Barnes, N. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00861 · Cited by 423
Composing Text and Image for Image Retrieval - an Empirical Odyssey
Vo, N., Jiang, L., Sun, C., Murphy, K., Li, L.-J., Fei-Fei, L., & Hays, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00660 · Cited by 399
More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch
Calandra, R., Owens, A., Jayaraman, D., Lin, J., Yuan, W., Malik, J., Adelson, E. H., & Levine, S. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2852779 · Cited by 359
Video Summarization Using Deep Neural Networks: A Survey
Apostolidis, E., Adamantidou, E., Metsai, A. I., Mezaris, V., & Patras, I. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3117472 · Cited by 350
Owens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E. H., & Freeman, W. T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.264 · Cited by 346
Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM
Zhu, G., Zhang, L., Shen, P., & Song, J. (2017) · IEEE Access · doi:10.1109/access.2017.2684186 · Cited by 270
A2dele: Adaptive and Attentive Depth Distiller for Efficient RGB-D Salient Object Detection
Piao, Y., Rong, Z., Zhang, M., Ren, W., & Lu, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00908 · Cited by 263
Adversarial Representation Learning for Text-to-Image Matching
Sarafianos, N., Xu, X., & Kakadiaris, I. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00591 · Cited by 250
Visual–Tactile Fusion for Object Recognition
Liu, H., Yu, Y., Sun, F., & Gu, J. (2017) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2016.2549552 · Cited by 248
Self-supervised Learning of Audio-Visual Objects from Video
Afouras, T., Owens, A., Chung, J. S., & Zisserman, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58523-5_13 · Cited by 217
Binding Touch to Everything: Learning Unified Multimodal Tactile Representations
Yang, F., Feng, C., Chen, Z., Park, H., Wang, D., Dou, Y., Zeng, Z., Chen, X., Gangopadhyay, S., Owens, A., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02488 · Cited by 214
Visual to Sound: Generating Natural Sound for Videos in the Wild
Zhou, Y., Wang, Z., Fang, C., Bui, T., & Berg, T. L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00374 · Cited by 201
Zhou, L., Kalantidis, Y., Chen, X., Corso, J. J., & Rohrbach, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00674 · Cited by 186
Deep Cross-Modal Audio-Visual Generation
Chen, L., Srivastava, S., Duan, Z., & Xu, C. (2017) · Proceedings of the on Thematic Workshops of ACM Multimedia 2017 · doi:10.1145/3126686.3126723 · Cited by 186
Wav2CLIP: Learning Robust Audio Representations from Clip
Wu, H.-H., Seetharaman, P., Kumar, K., & Bello, J. P. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747669 · Cited by 182
Multi-modal gesture recognition challenge 2013: dataset and results
Escalera, S., Gonzàlez, J., Baró, X., Reyes, M., Lopes, O., Guyon, I., Athitsos, V., & Escalante, H. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2532595 · Cited by 177
Connecting Look and Feel: Associating the Visual and Tactile Properties of Physical Materials
Yuan, W., Wang, S., Dong, S., & Adelson, E. H. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.478 · Cited by 167
Weakly Supervised Dense Video Captioning
Shen, Z., Li, J., Su, Z., Li, M., Chen, Y., Jiang, Y.-G., & Xue, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.548 · Cited by 141
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
Feng, W., Zhu, W., Fu, T.-J., Jampani, V., Akula, A., He, X., Basu, S., Wang, X. E., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.15393 · Cited by 140
A Touch, Vision, and Language Dataset for Multimodal AlignmentPreprint
Fu, L., Datta, G., Huang, H., Panitch, W. C.-H., Drake, J., Ortiz, J., Mukadam, M., Lambeta, M., Calandra, R., & Goldberg, K. Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.13232 · Cited by 138
Lee, J.-T., Bollegala, D., & Luo, S. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793763 · Cited by 128
Generating Visually Aligned Sound From Videos
Chen, P., Zhang, Y., Tan, M., Xiao, H., Huang, D. Y., & Gan, C. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3009820 · Cited by 126
Enhancing visual perception of shape through tactile glances
Björkman, M., Bekiroglu, Y., Högman, V., & Kragić, D. (2013) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2013.6696808 · Cited by 126
Improving Automatic VQA Evaluation Using Large Language Models
Mañas, O., Krojer, B., & Agrawal, A. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28212 · Cited by 123
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models
Cheng, A.-C., Yin, H., Fu, Y., Guo, Q., Yang, R., Kautz, J., Wang, X., & Liu, S. (2024) · arXiv · doi:10.48550/arxiv.2406.01584 · Cited by 121
SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models
Cheng, A.-C., Yin, H., Fu, Y., Guo, Q., Yang, R., Kautz, J., Wang, X., & Liu, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-4293 · Cited by 121
Enhancing Video Summarization via Vision-Language Embedding
Plummer, B. A., Brown, M., & Lazebnik, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.118 · Cited by 113
Proceedings of the 2018 International Symposium on Experimental Robotics
(2020) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-33950-0 · Cited by 110
Im2Flow: Motion Hallucination from Static Images for Action Recognition
Gao, R., Xiong, B., & Grauman, K. L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00622 · Cited by 105
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models
Liu, Y., Zhang, K., Li, Y., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y., Sun, H., Gao, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.17177 · Cited by 102
Hierarchical multimodal transformer to summarize videos
Zhao, B., Gong, M., & Li, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.10.039 · Cited by 101
Self-citationProceedings of the 2020 International Conference on Multimodal Interaction
Truong, K. P., Heylen, D., Czerwinski, M. P., Bianchi‐Berthouze, N., Chétouani, M., & Nakano, M. (2020) · doi:10.1145/3382507 · Cited by 100
Event-Stream Representation for Human Gaits Identification Using Deep Neural Networks
Wang, Y., Zhang, X., Shen, Y., Du, B., Zhao, G., Lizhen, L. C. C., & Wen, H. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3054886 · Cited by 94
Self-Attention Based Visual-Tactile Fusion Learning for Predicting Grasp Outcomes
Cui, S., Wang, R., Wei, J., Hu, J., & Wang, S. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3010720 · Cited by 91
Grasp State Assessment of Deformable Objects Using Visual-Tactile Fusion Perception
Cui, S., Wang, R., Wei, J., Li, F., & Wang, S. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196787 · Cited by 89
CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios
Ye, Q., Yu, Z., Shao, R., Xie, X., Torr, P. H. S., & Cao, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72684-2_9 · Cited by 85
Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images
Takahashi, K., & Tan, J. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794285 · Cited by 85
EVA-CLIP: Improved Training Techniques for CLIP at Scale
Sun, Q., Fang, Y., Wu, L., Wang, X., & Cao, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.15389 · Cited by 79
AudioVisual Video Summarization
Zhao, B., Gong, M., & Li, X. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3119969 · Cited by 78
Self-citationQuery-controllable Video Summarization
Huang, J.-H., & Worring, M. (2020) · International Conference on Multimedia Retrieval · doi:10.1145/3372278.3390695 · Cited by 76
Spectral Multimodal Hashing and Its Application to Multimedia Retrieval
Zhen, Y., Gao, Y., Yeung, D.-Y., Zha, H., & Li, X. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2392052 · Cited by 73
Proceedings of the 2022 International Conference on Multimedia Retrieval
(2022) · doi:10.1145/3512527 · Cited by 71
Browatzki, B., Fischer, J., Graf, B., Bulthoff, H. H., & Wallraven, C. (2011) · 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) · doi:10.1109/iccvw.2011.6130385 · Cited by 71
TAFNet: A Three-Stream Adaptive Fusion Network for RGB-T Crowd Counting
Tang, H., Wang, Y., & Chau, L. (2022) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas48785.2022.9937583 · Cited by 68
Learning Effective RGB-D Representations for Scene Recognition
Song, X., Jiang, S., Herranz, L., & Chen, C. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2872629 · Cited by 67
A Multi-Modal, Discriminative and Spatially Invariant CNN for RGB-D Object Labeling
Asif, U., Bennamoun, M., & Sohel, F. A. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2747134 · Cited by 66
Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications
Senocak, A., Oh, T.-H., Kim, J., Yang, M., & Kweon, I. S. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2952095 · Cited by 60
Proceedings of the on Thematic Workshops of ACM Multimedia 2017
Wu, W., Yan, S., Tian, Q., & Zimmermann, R. (2017) · doi:10.1145/3126686 · Cited by 59
Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation
Yariv, G., Gat, I., Benaim, S., Wolf, L., Schwartz, I., & Adi, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28486 · Cited by 59
Gesture recognition based on multilevel multimodal feature fusion
Tian, J., Cheng, W., Sun, Y., Li, G., Jiang, D., Jiang, G., Tao, B., Zhao, H., & Chen, D. (2012) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-179541 · Cited by 57
Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
Lin, J., Calandra, R., & Levine, S. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793885 · Cited by 57
Lifelong Visual-Tactile Cross-Modal Learning for Robotic Material Perception
Zheng, W., Liu, H., & Sun, F. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2980892 · Cited by 52
iQuery: Instruments as Queries for Audio-Visual Sound Separation
Chen, J., Zhang, R., Lian, D., Yang, J., Zeng, Z., & Shi, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01410 · Cited by 48
Visual-Tactile Learning of Garment Unfolding for Robot-Assisted Dressing
Zhang, F., & Demiris, Y. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3296371 · Cited by 47
Falco, P., Lu, S., Natale, C., Pirozzi, S., & Lee, D. (2019) · IEEE Transactions on Robotics · doi:10.1109/tro.2019.2914772 · Cited by 44
ASK: Adaptively Selecting Key Local Features for RGB-D Scene Recognition
Xiong, Z., Yuan, Y., & Wang, Q. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3053459 · Cited by 44
Self-citationEncoder deep interleaved network with multi-scale aggregation for RGB-D salient object detection
Feng, G., Meng, J., Zhang, L., & Lu, H. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108666 · Cited by 39
VITO-Transformer: A Visual-Tactile Fusion Network for Object Recognition
Li, B., Bai, J., Qiu, S., Wang, H., & Guo, Y. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3326241 · Cited by 39
Large-Scale Multimodal Gesture Segmentation and Recognition Based on Convolutional Neural Networks
Wang, H., Wang, P., Song, Z., & Li, W. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.371 · Cited by 37
Depth Injection Framework for RGBD Salient Object Detection
Yao, S., Zhang, M., Piao, Y., Qiu, C., & Lu, H. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3315511 · Cited by 37
Semi-Supervised Cross-Modality Action Recognition by Latent Tensor Transfer Learning
Jia, C., Ding, Z., Kong, Y., & Fu, Y. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2910208 · Cited by 34
Video summarization with u-shaped transformer
Chen, Y., Guo, B., Shen, Y., Zhou, R., Lu, W., Wang, W., Wen, X., & Suo, X. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03451-1 · Cited by 34
Fine-grained Audio-Visual Joint Representations for Multimodal Large Language ModelsPreprint
Sun, G., Yu, W., Tang, C., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.05863 · Cited by 34
Video summarization via knowledge-aware multimodal deep networks
Xie, J., Chen, X., Zhao, S., & Lu, S.-P. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111670 · Cited by 31
RigidFusion: RGB‐D Scene Reconstruction with Rigidly‐moving Objects
Wong, Y., Li, C., Nießner, M., & Mitra, N. J. (2021) · Computer Graphics Forum · doi:10.1111/cgf.142651 · Cited by 31
Visuo-Tactile Feedback-Based Robot Manipulation for Object Packing
Liang, W., Fang, F., Acar, C., Toh, W. Q., Sun, Y., Xu, Q., & Wu, Y. J. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3236884 · Cited by 30
Boosting RGB-D Saliency Detection by Leveraging Unlabeled RGB Images
Wang, X., Zhu, L., Tang, S., Fu, H., Li, P., Wu, F. Q., Yang, Y., & Zhuang, Y. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3139232 · Cited by 29
Localize to Binauralize: Audio Spatialization from Visual Sound Source Localization
Rachavarapu, K. K., Aakanksha, A., Sundaresha, V., & Rajagopalan, A. N. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00194 · Cited by 27
Multi-stage Factorized Spatio-Temporal Representation for RGB-D Action and Gesture Recognition
Ma, Y., Zhou, B., Wang, R., & Wang, P. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612301 · Cited by 26
Proceedings 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering
(2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00002 · Cited by 24
Joint Inference of Kinematic and Force Trajectories with Visuo-Tactile Sensing
Lambert, A. S., Mukadam, M., Sundaralingam, B., Ratliff, N., Boots, B., & Fox, D. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794048 · Cited by 23
Binaural audio generation via multi-task learning
Li, S., Liu, S., & Manocha, D. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480560 · Cited by 22
Seamless Texture Optimization for RGB-D Reconstruction
Fu, Y., Yan, Q., Liao, J., Zhou, H., Tang, J., & Xiao, C. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3134105 · Cited by 21
VG4D: Vision-Language Model Goes 4D Video Recognition
Deng, Z., Li, X., Li, X., Tong, Y., Zhao, S., & Liu, M. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610217 · Cited by 21
Audio-Visual Contrastive and Consistency Learning for Semi-Supervised Action Recognition
Assefa, M., Jiang, W., Zhan, J., Gedamu, K., Yilma, G., Ayalew, M., & Adhikari, D. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3312856 · Cited by 19
Detection Based Part-level Articulated Object Reconstruction from Single RGBD Image
Kawana, Y., & Harada, T. (2023) · neural information processing systems · doi:10.52202/075280-0811 · Cited by 19
Li, T., Sun, Z., & Xiao, X. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3372469 · Cited by 18
Kunitsyn, A., Kalashnikov, M., Dzabraev, M., & Ivaniuta, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.07086 · Cited by 17
TaoHighlight: Commodity-Aware Multi-Modal Video Highlight Detection in E-Commerce
Guo, Z., Zhao, Z., Jin, W., Wang, D., Liu, R., & Yu, J. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3087001 · Cited by 13
CoEvo-Net: Coevolution Network for Video Highlight Detection
Chen, J., Wang, J., Wang, X., Wang, X., Feng, Z., Liu, R., & Song, M. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3113505 · Cited by 10
Sora as a World Model? A Complete Survey on Text-to-Video Generation
Puspitasari, F. D., Zhang, C., Cho, J., Haider, A., Eman, N. U., Amin, O., Mankowski, A., Umair, M., Zheng, J., Zheng, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05131 · Cited by 10
A PCA–CCA network for RGB-D object recognition
Sun, S., An, N., Zhao, X., & Tan, M. H. (2018) · International Journal of Advanced Robotic Systems · doi:10.1177/1729881417752820 · Cited by 9
Multi-Attention Audio-Visual Fusion Network for Audio Spatialization
Zhang, W., & Shao, J. (2021) · International Conference on Multimedia Retrieval · doi:10.1145/3460426.3463624 · Cited by 7
Lee, H., & Lee, G. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9190636 · Cited by 6
Summarizing Relevant Parts from Technical Videos
Vahedi, M., Rahman, M. M., Khomh, F., Uddin, G., & Antoniol, G. (2021) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner50967.2021.00047 · Cited by 6
Multi-scale Cross-Modal Transformer Network for RGB-D Object Detection
Xiao, Z., Xie, P., & Wang, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98358-1_28 · Cited by 5
Bio-Inspired Audiovisual Multi-Representation Integration via Self-Supervised Learning
Li, Z., Zhao, B., & Yuan, Y. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612428 · Cited by 5
Self-citationA Human-Like Siamese-Based Visual-Tactile Fusion Model for Object Recognition
Wang, F., Li, Y., Tao, L., Wu, J., & Huang, G. (2023) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2023.3311871 · Cited by 4
SpatialBot: Precise Spatial Understanding with Vision Language ModelsPreprint
Cai, W., Ponomarenko, I., Yuan, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.13642 · Cited by 2
Progressive Video Summarization via Multimodal Self-supervised LearningPreprint
Haopeng, L., Ke, Q., Gong, M., & Drummond, T. W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.02494 · Cited by 2
iQuery: Instruments as Queries for Audio-Visual Sound SeparationPreprint
Chen, J., Zhang, R., Lian, D., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.03814 · Cited by 1
A Survey on Multimodal Large Language Models
Yin, S., Fu, C., Zhao, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.13549
Weakly Supervised Visual-Auditory Fixation Prediction with Multigranularity Perception
Wang, G., Chen, C., Fan, D.-P., Hao, A., & Qin, H. (2021) · arXiv · doi:10.48550/arxiv.2112.13697
A supervised learning approach for fast object recognition from RGB-D data
Paulk, D., Metsis, V., McMurrough, C., & Makedon, F. (2014) · Proceedings of the 7th International Conference on PErvasive Technologies Related to Assistive Environments · doi:10.1145/2674396.2674432
Research works printed but not linked: 56
- Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr Yana Hasson Karel Lenc Arthur Mensch Katherine Millican Malcolm Reynolds et al. 2022. Flamingo: A visual language model for few-shot learning. Advances in Neural Information Processing Systems 35 (2022) 23716–23736.
- Dawit Mureja Argaw, Seunghyun Yoon, Fabian Caba Heilbron, Hanieh Deilamsalehy, Trung Bui, Zhaowen Wang, Franck Dernoncourt, and Joon Son Chung. 2024. Scaling up video summarization pretraining with large language models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8332–8341.
- Taivanbat Badamdorj, Mrigank Rochan, Yang Wang, and Li Cheng. 2021. Joint visual and audio learning for video highlight detection. In Proceedings of the IEEEF International Conference on Computer Vision. 8127–8137.
- Maxime Burchi and Radu Timofte. 2023. Audio-visual efficient conformer for robust speech recognition. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 2258–2267.
- Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2022. Zero-shot audio source separation through query-based learning from weakly-labeled data. In Proceedings of the Association for the Advance of Artificial Intelligence. 4441–4449.
- Angela Dai, Christian Diller, and Matthias Nießner. 2020. SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 849–858.
- Danny Driess Fei Xia Mehdi S. M. Sajjadi Corey Lynch Aakanksha Chowdhery Brian Ichter Ayzaan Wahid Jonathan Tompson Quan Vuong Tianhe Yu Wenlong Huang Yevgen Chebotar Pierre Sermanet Daniel Duckworth Sergey Levine Vincent Vanhoucke Karol Hausman Marc Toussaint Klaus Greff Andy Zeng Igor Mordatch and Pete Florence. 2023. PaLM-E: An embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning. 8469–8488.
- Ruihan Gao, Wenzhen Yuan, and Jun-Yan Zhu. 2023. Controllable visual-tactile synthesis. In Proceedings of the IEEE International Conference on Computer Vision. 7040–7052.
- Mallika Garg, Debashis Ghosh, and Pyari Mohan Pradhan. 2024. GestFormer: Multiscale wavelet pooling transformer network for dynamic hand gesture recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2473–2483.
- Rishabh Garg Ruohan Gao and Kristen Grauman. 2021. Geometry-aware multi-task learning for binaural audio generation from video. In Proceedings of the 32nd British Machine Vision Conference. 1.
- Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. ImageBind: One embedding space to bind them all. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 15180–15190.
- Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. 2023. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10867–10877.
- D. Hu, D. Wang, X. Li, F. Nie, and Q. Wang. 2019. Listen to the image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7964–7973.
- Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson W. H. Lau, Wanli Ouyang, and Wangmeng Zuo. 2023. CLIP2Point: Transfer CLIP to point cloud classification with image-depth pre-training. In Proceedings of the IEEE International Conference on Computer Vision. 22157–22167.
- Muhammad Rafiqul Islam, Manoranjan Paul, Michael Antolovich, and Ashad Kabir. 2019. Sports highlights generation using decomposed audio information. In IEEE International Conference on Multimedia & Expo Workshops. 579–584.
- Vahid Kiani and Hamid Reza Pourreza. 2013. Flexible soccer video summarization in compressed domain. In Proceedings of the IEEE International Conference on Computer and Knowledge Engineering. 213–218.
- Jing Yu Koh Daniel Fried and Russ R. Salakhutdinov. 2023. Generating images with multimodal language models. Advances in Neural Information Processing Systems 36 (2023) 21487–21506.
- Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. 2022. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 19108–19118.
- Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning. 12888–12900.
- Yunan Li, Huizhou Chen, Guanwen Feng, and Qiguang Miao. 2023. Learning robust representations with information bottleneck and memory network for RGB-D-based gesture recognition. In Proceedings of the IEEE International Conference on Computer Vision. 20968–20978.
- Y. Li, J. Zhu, R. Tedrake, and A. Torralba. 2019. Connecting touch and vision via cross-modal prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10601–10610.
- Zhaojian Li, Bin Zhao, and Yuan Yuan. 2024. Cyclic learning for binaural audio generation and localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 26669–26678.
- Miao Liu, Jing Wang, Xinyuan Qian, and Xiang Xie. 2024. Visually guided binaural audio generation with cross-modal consistency. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. 7980–7984.
- Yujie Lu Xianjun Yang Xiujun Li Xin Eric Wang and William Yang Wang. 2023. LLMScore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems 36 (2023) 23075–23093.
- Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2021. Thinking fast and slow: Efficient text-to-visual retrieval with transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9826–9836.
- Medhini Narasimhan Anna Rohrbach and Trevor Darrell. 2021. Clip-It! Language-guided video summarization. Advances in Neural Information Processing Systems 34 (2021) 13988–14000.
- Anshul Paigwar, David Sierra-Gonzalez, Özgür Erkent, and Christian Laugier. 2021. Frustum-pointpillars: A multi-stage approach for 3D object detection using rgb camera and lidar. In Proceedings of the IEEE International Conference on Computer Vision. 2926–2933.
- Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. 2022. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8238–8247.
- J. Perez-Rua, V. Vielzeuf, S. Pateux, M. Baccouche, and F. Jurie. 2019. MFAS: Multimodal fusion architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6959–6968.
- Jens Piekenbrinck, Alexander Hermans, Narunas Vaskevicius, Timm Linder, and Bastian Leibe. 2024. RGB-D cube R-CNN: 3D object detection with selective modality dropout. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1997–2006.
- Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. In Proceedings of International Conference on Machine Learning. 8821–8831.
- Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016. Generative adversarial text to image synthesis. In Proceedings of the International Conference on Machine Learning. 1060–1069.
- Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. 2024. TimeChat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 14313–14323.
- Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Besmira Nushi, and Ece Kamar. 2020. SQuINTing at VQA Models: Introspecting VQA models with sub-questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10003–10011.
- Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 14974–14983.
- M. Subedar, R. Krishnan, P. L. Meyer, O. Tickoo, and J. Huang. 2019. Uncertainty-aware audiovisual activity recognition using deep bayesian variational inference. In Proceedings of the IEEE International Conference on Computer Vision. 6300–6309.
- Weixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu, Yiran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. 2023. Learning audio-visual source localization via false negative aware contrastive learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6420–6429.
- Pichao Wang, Wanqing Li, Jun Wan, Philip Ogunbona, and Xinwang Liu. 2018. Cooperative training of deep aggregation networks for RGB-D action recognition. In Proceedings of the Association for the Advance of Artificial Intelligence. 7404–7411.
- S. Wang, J. Wu, X. Sun, W. Yuan, W. T. Freeman, J. B. Tenenbaum, and E. H. Adelson. 2018. 3D shape perception from monocular vision, touch, and shape priors. In Proceedings of the IEEE International Conference on Intelligent Robots and Systems. 1606–1613.
- W. Wang, D. Tran, and M. Feiszli. 2020. What makes training multi-modal classification networks hard?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12692–12702.
- Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and Liangwei Wang. 2020. On the general value of evidence, and bilingual scene-text visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10126–10135.
- Zhengwei Wang, Qi She, Tejo Chalasani, and Aljosa Smolic. 2020. CatNet: Class incremental 3D ConvNets for lifelong egocentric gesture recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 935–944.
- Y. Wu, L. Zhu, Y. Yan, and Y. Yang. 2019. Dual attention matching for audio-visual event localization. In Proceedings of the IEEE International Conference on Computer Vision. 6291–6299.
- Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. 2024. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7151–7161.
- Wenqiang Xu, Zhenjun Yu, Han Xue, Ruolin Ye, Siqiong Yao, and Cewu Lu. 2023. Visual-tactile sensing for in-hand object reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8803–8812.
- Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023. Vid2Seq: Large-scale pretraining of a visual language model for dense video captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10714–10726.
- Erkun Yang, Cheng Deng, Wei Liu, Xianglong Liu, Dacheng Tao, and Xinbo Gao. 2017. Pairwise relationship guided deep hashing for cross-modal retrieval. In Proceedings of the Association for the Advance of Artificial Intelligence. 1618–1625.
- Fengyu Yang, Jiacheng Zhang, and Andrew Owens. 2023. Generating visual scenes from touch. In Proceedings of the IEEE International Conference on Computer Vision. 22070–22080.
- K. Yang, B. Russell, and J. Salamon. 2020. Telling left from right: Learning spatial correspondence of sight and sound. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9929–9938.
- Qinghao Ye, Xiyue Shen, Yuan Gao, Zirui Wang, Qi Bi, Ping Li, and Guang Yang. 2021. Temporal cue guided video highlight detection with low-rank audio-visual fusion. In Proceedings of the IEEE International Conference on Computer Vision. 7950–7959.
- M. Zhang, W. Ren, Y. Piao, Z. Rong, and H. Lu. 2020. Select, supplement and focus for RGB-D saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3469–3478.
- H. Zhao, C. Gan, W. Ma, and A. Torralba. 2019. The sound of motions. In Proceedings of the IEEE International Conference on Computer Vision. 1735–1744.
- Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh H. McDermott, and Antonio Torralba. 2018. The sound of pixels. In Proceedings of the European Conference on Computer Vision. 587–604.
- Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. 2020. Sep-Stereo: Visually guided stereophonic audio generation by associating source separation. In Proceedings of the European Conference on Computer Vision. 52–69.
- Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, and Hanwang Zhang. 2020. More grounded image captioning by distilling image-text matching model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4776–4785.
- Lingyu Zhu and Esa Rahtu. 2022. Visually guided sound source separation and localization using self-supervised motion representations. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 1289–1299.
Cited by
Aligning generalization between humans and machines
Ilievski, F., Hammer, B., van Harmelen, F., Paaßen, B., Saralajew, S., Schmid, U., Biehl, M. L., Bolognesi, M., Dong, X., Gashteovski, K., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01109-4 · Cited by 58
Sun, J., Khalid, K. A. T., & Kay, C. S. (2025) · Future Technology · doi:10.55670/fpll.futech.4.3.12 · Cited by 26
He, Y., & Zhang, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-14164-z · Cited by 28
Combining Wearable Sensors and Artificial Intelligence in Cardiovascular Medicine
Maiorana, E., Campisi, P., Schena, E., & Massaroni, C. (2025) · IEEE Sensors Reviews · doi:10.1109/sr.2025.3623069 · Cited by 15
Large Language and Foundation Models for Machinery Health Monitoring: A Systematic Review
Tsallis, C., Papageorgas, P. G., Munteanu, R. A., & Dellagi, S. (2026) · Applied Sciences · doi:10.3390/app16052493 · Cited by 10
Metadata
- Authors
- Yuan, Y., Li, Z., & Zhao, B.
- Affiliations
- Northwestern Polytechnical University (CN)
- Venue
- ACM Computing Surveys
- Published
- 2025-02-20
- Volume
- 57
- Issue
- 7
- Pages
- 1-34
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4406552230
- Semantic Scholar
- 1f31434605e5db7f8033149ab0b713fc6829d67d
- Cited by
- 90
- References
- 82
- Citations by year
- 2026: 69 · 2025: 21
- FWCI
- 31.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Computing Surveys (publishedVersion)
- Funders
- National Science Fund for Distinguished Young Scholars (61825603)
- National Natural Science Foundation of China (62106183)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, cross-modal relationships, multimodal dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.