Multimodal Co-learning: Challenges, applications with datasets, recent advances and future directions

Deep Residual Learning for Image Recognition

He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works

Attention Is All You Need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works

MizAR 60 for Mizar 50

Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works

Very Deep Convolutional Networks for Large-Scale Image Recognition

Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works

Gradient-based learning applied to document recognition

Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works

You Only Look Once: Unified, Real-Time Object Detection

Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · see citing works

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works

ImageNet Large Scale Visual Recognition Challenge

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works

Glove: Global Vectors for Word Representation

Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works

SMOTE synthetic minority over-sampling technique

Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · see citing works

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works

Distributed Representations of Words and Phrases and their Compositionality

Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works

Learning Multiple Layers of Features from Tiny Images

Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works

A survey on transfer learning

Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works

Language Models are Few-Shot Learners

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works

Generative Adversarial Nets

Labaca-Castro, R. (2023) · Springer eBooks · doi:10.1007/978-3-658-40442-0_9 · Cited by 21,100 · see citing works

GAN（Generative Adversarial Nets）

淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,268 · see citing works

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works

Learning Transferable Visual Models From Natural Language Supervision

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works

Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics

Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works

Momentum Contrast for Unsupervised Visual Representation Learning

He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works

Deep Contextualized Word Representations

Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works

WordNet: An electronic lexical database . Ed. by Christiane Fellbaum. Cambridge, MA: MIT Press, 1998. Pp. xxii, 423.

Kilgarriff, A. (2000) · Language · doi:10.2307/417141 · Cited by 11,595 · see citing works

A simple framework for contrastive learning of visual representations

Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works

Conditional Generative Adversarial Nets

Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863 · see citing works

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works

Multitask Learning

Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636 · see citing works

A Survey on Bias and Fairness in Machine Learning

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · see citing works

Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network

Sherstinsky, A. (2020) · Physica D: Nonlinear Phenomena · doi:10.1016/j.physd.2019.132306 · Cited by 5,680 · see citing works

Combining labeled and unlabeled data with co-training

Blum, A. L., & Mitchell, T. M. (1998) · Conference on Learning Theory · doi:10.1145/279943.279962 · Cited by 5,597 · see citing works

Multimodal Machine Learning: A Survey and Taxonomy

Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works

ADASYN: Adaptive synthetic sampling approach for imbalanced learning

He, H., Bai, Y., Garcia, E. A., & Shutao Li, A. (2008) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2008.4633969 · Cited by 4,945 · see citing works

Hierarchical Attention Networks for Document Classification

Yang, Z., Yang, D., Dyer, C., He, X., Smola, A. J., & Hovy, E. H. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1174 · Cited by 4,874 · see citing works

Reading digits in natural images with unsupervised feature learning

Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,261 · see citing works

IEMOCAP: Interactive emotional dyadic motion capture database

Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · see citing works

Unsupervised Feature Learning via Non-parametric Instance Discrimination

Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664 · see citing works

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial NetsPreprint

Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.03657 · Cited by 3,379 · see citing works

Generalizing from a Few Examples: A Survey on Few-shot Learning

Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2021) · ACM Computing Surveys · doi:10.1145/3386252 · Cited by 3,165 · see citing works

One-shot learning of object categories

Fei-Fei, L., Fergus, R., & Perona, P. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.79 · Cited by 3,029 · see citing works

Conditional Image Synthesis With Auxiliary Classifier GANsPreprint

Odena, A., Olah, C., & Shlens, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09585 · Cited by 2,968 · see citing works

Optimization as a Model for Few-Shot Learning

Ravi, S., & Larochelle, H. (2017) · International Conference on Learning Representations · Cited by 2,766 · see citing works

Untitled work

(2011) · Cited by 2,752 · see citing works

Untitled work

(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534 · see citing works

A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI

Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524 · see citing works

Explaining Explanations: An Overview of Interpretability of Machine Learning

Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · see citing works

Generative Adversarial Text to Image Synthesis

Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494 · see citing works

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · see citing works

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · see citing works

DeViSE: A Deep Visual-Semantic Embedding Model

Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,309 · see citing works

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · see citing works

Tensor Fusion Network for Multimodal Sentiment Analysis

Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · see citing works

Contrastive Multiview Coding

Tian, Y., Krishnan, D., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_45 · Cited by 1,997 · see citing works

Deep Boltzmann machines

Salakhutdinov, R., & Hinton, G. E. (2009) · Cited by 1,890 · see citing works

Classification in the Presence of Label Noise: A Survey

Frenay, B., & Verleysen, M. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2292894 · Cited by 1,777 · see citing works

Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy LabelsPreprint

Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I. W.-H., & Sugiyama, M. (2018) · arXiv (Cornell University) · doi:10.5555/3327757.3327944 · Cited by 1,674 · see citing works

A review of affective computing: From unimodal analysis to multimodal fusion

Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · see citing works

Contrastive Learning for Unpaired Image-to-Image Translation

Park, T., Efros, A. A., Zhang, R., & Zhu, J.-Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58545-7_19 · Cited by 1,605 · see citing works

An Introduction to Convolutional Neural NetworksPreprint

O’Shea, K., & Nash, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.08458 · Cited by 1,463 · see citing works

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Hendrycks, D., & Dietterich, T. (2019) · arXiv · doi:10.48550/arxiv.1903.12261 · Cited by 1,335 · see citing works

Domain Generalization: A Survey

Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3195549 · Cited by 1,315 · see citing works

Hierarchical Question-Image Co-Attention for Visual Question AnsweringPreprint

Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,303 · see citing works

Interpretable Machine Learning: Fundamental Principles and 10 Grand ChallengesPreprint

Rudin, C., Chen, C.-F., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2021) · Statistics Survey · doi:10.48550/arxiv.2103.11251 · Cited by 1,254 · see citing works

VideoBERT: A Joint Model for Video and Language Representation Learning

Sun, C., Myers, A., Vondrick, C., Murphy, K. P., & Schmid, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,212 · see citing works

VisualBERT: A Simple and Performant Baseline for Vision and Language

Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · see citing works

Efficient Low-rank Multimodal Fusion With Modality-Specific Factors

Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170 · see citing works

Multi-view learning overview: Recent progress and new challenges

Zhao, J., Xie, X., Xu, X., & Sun, S. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.007 · Cited by 1,002 · see citing works

Zero-Shot Learning Through Cross-Modal Transfer

Socher, R., Ganjoo, M., Sridhar, H., Bastani, O., Manning, C. D., & Ng, A. Y. (2013) · arXiv · doi:10.48550/arxiv.1301.3666 · Cited by 997 · see citing works

Translating Videos to Natural Language Using Deep Recurrent Neural Networks

Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R. J., & Saenko, K. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1173 · Cited by 996 · see citing works

Learning from massive noisy labeled data for image classification

Xiao, T., Xia, T., Yang, Y., Huang, C., & Wang, X. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298885 · Cited by 971 · see citing works

Interpretable machine learning: Fundamental principles and 10 grand challenges

Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958 · see citing works

“Put-that-there”

Bolt, R. A. (1980) · ACM SIGGRAPH Computer Graphics · doi:10.1145/965105.807503 · Cited by 921 · see citing works

Meta-Learning in Neural Networks: A Survey

Hospedales, T. M., Antoniou, A., Micaelli, P., & Storkey, A. J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3079209 · Cited by 886 · see citing works

A survey of multi-view machine learning

Sun, S. (2013) · Neural Computing and Applications · doi:10.1007/s00521-013-1362-6 · Cited by 878 · see citing works

MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted LabelsPreprint

Jiang, L., Zhou, Z., Leung, T., Li, L.-J., & Fei-Fei, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.05055 · Cited by 834 · see citing works

A survey on deep learning for multimodal data fusion

Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821 · see citing works

Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review

Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806 · see citing works

A Comprehensive Survey on Domain Adaptation for Visual Applications

Csurka, G. (2017) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-319-58347-1_1 · Cited by 774 · see citing works

Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions

Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771 · see citing works

Towards Automatic Learning of Procedures From Web Instructional Videos

Zhou, L., Xu, C., & Corso, J. J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12342 · Cited by 663 · see citing works

SoundNet: Learning Sound Representations from Unlabeled VideoPreprint

Aytar, Y., Vondrick, C., & Torralba, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09001 · Cited by 658 · see citing works

Combating Noisy Labels by Agreement: A Joint Training Method with Co-Regularization

Wei, H., Feng, L., Chen, X., & An, B. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01374 · Cited by 649 · see citing works

GRAM: Graph-based Attention Model for Healthcare Representation Learning

Choi, E., Bahadori, M. T., Song, L., Stewart, W. F., & Sun, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098126 · Cited by 637 · see citing works

End-to-End Learning of Visual Representations From Uncurated Instructional Videos

Miech, A., Alayrac, J.-B., Smaira, L., Laptev, I., Šivic, J., & Zisserman, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00990 · Cited by 622 · see citing works

What Makes Training Multi-modal Classification Networks Hard?

Wang, W., Tran, D., & Feiszli, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01271 · Cited by 588 · see citing works

Jointly Modeling Embedding and Translation to Bridge Video and Language

Pan, Y., Mei, T., Yao, T., Li, H., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.497 · Cited by 573 · see citing works

Cross-modal Retrieval with Correspondence Autoencoder

Feng, F., Wang, X., & Li, R. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654902 · Cited by 572 · see citing works

Multiple instance classification: Review, taxonomy and comparative study

Amores, J. (2013) · Artificial Intelligence · doi:10.1016/j.artint.2013.06.003 · Cited by 562 · see citing works

Deep Co-Training for Semi-Supervised Image Recognition

Qiao, S., Shen, W., Zhang, Z., Wang, B., & Yuille, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_9 · Cited by 538 · see citing works

How does Disagreement Help Generalization against Label Corruption?Preprint

Yu, X., Han, B., Yao, J., Niu, G., Tsang, I. W.-H., & Sugiyama, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.04215 · Cited by 538 · see citing works

Deep Multimodal Representation Learning: A Survey

Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535 · see citing works

A Survey of Zero-Shot Learning: Settings, Methods, and Applications

Wang, W., Zheng, V. W., Yu, H., & Miao, C. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3293318 · Cited by 528 · see citing works

Scene Graph Generation from Objects, Phrases and Region Captions

Li, Y., Ouyang, W., Zhou, B., Wang, K., & Wang, X. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.142 · Cited by 518 · see citing works

Found in Translation: Learning Robust Joint Representations by Cyclic Translations between Modalities

Pham, H., Liang, P. P., Manzini, T., Morency, L.-P., & Póczos, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016892 · Cited by 512 · see citing works

Multimodal intelligence: Representation learning, information fusion, and applications

Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511 · see citing works

Contrastive Multiview Coding

Tian, Y., Krishnan, D., & Isola, P. (2019) · arXiv · doi:10.48550/arxiv.1906.05849 · Cited by 465 · see citing works

Counterfactual VQA: A Cause-Effect Look at Language Bias

Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X., & Wen, J.-R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01251 · Cited by 448 · see citing works

Untitled work

(2008) · Cited by 436 · see citing works

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and TextPreprint

Akbari, H., Yuan, L., Qian, R., Chuang, W.-H., Chang, S., Cui, Y., & Gong, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.11178 · Cited by 433 · see citing works

Domain Adaptation in Computer Vision Applications

Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-319-58347-1 · Cited by 430 · see citing works

Multi-Task Learning with Deep Neural Networks: A Survey

Crawshaw, M. (2020) · arXiv · doi:10.48550/arxiv.2009.09796 · Cited by 428 · see citing works

Multimodal semi-supervised learning for image classification

Guillaumin, M., Verbeek, J. J., & Schmid, C. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540120 · Cited by 425 · see citing works

12-in-1: Multi-Task Vision and Language Representation Learning

Lu, J., Goswami, V., Rohrbach, M., Parikh, D., & Lee, S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01045 · Cited by 410 · see citing works

Billion-scale semi-supervised learning for image classificationPreprint

Yalnız, İ. Z., Jeǵou, H., Chen, K., Paluri, M., & Mahajan, D. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.00546 · Cited by 387 · see citing works

Multimodal sentiment analysis using hierarchical fusion with context modeling

Majumder, N., Hazarika, D., Gelbukh, A. F., Cambria, E., & Poria, S. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.07.041 · Cited by 385 · see citing works

An Overview of Deep Semi-Supervised LearningPreprint

Ouali, Y., Hudelot, C., & Tami, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05278 · Cited by 381 · see citing works

MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels

Jiang, L., Zhou, Z., Leung, T., Li, L.-J., & Fei-Fei, L. (2018) · International Conference on Machine Learning · Cited by 371 · see citing works

Mental health monitoring with multimodal sensing and machine learning: A survey

Garcia-Ceja, E., Riegler, M., Nordgreen, T., Jakobsen, P., Oedegaard, K. J., & Tørresen, J. (2018) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2018.09.003 · Cited by 356 · see citing works

WebVision Database: Visual Learning and Understanding from Web DataPreprint

Li, W., Wang, L., Wei, L., Agustsson, E., & Gool, L. V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02862 · Cited by 354 · see citing works

MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos

Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · arXiv · doi:10.48550/arxiv.1606.06259 · Cited by 350 · see citing works

Zero-data learning of new tasks

Larochelle, H., Erhan, D., & Bengio, Y. (2008) · Cited by 345 · see citing works

SMIL: Multimodal Learning with Severely Missing Modality

Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., & Peng, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16330 · Cited by 332 · cited 7× · see citing works

Learning Factorized Multimodal Representations

Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., & Salakhutdinov, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06176 · Cited by 323 · see citing works

Deep Learning Human Mind for Automated Visual Classification

Spampinato, C., Palazzo, S., Kavasidis, I., Giordano, D., Souly, N., & Shah, M. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.479 · Cited by 309 · see citing works

A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines

Charte, D., Charte, F., García, S., del Jesus, M. J., & Herrera, F. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.12.007 · Cited by 305 · see citing works

Gated Multimodal Units for Information FusionPreprint

Arévalo, J., Solorio, T., Montes-y-Gómez, M., & Gonzalez, F. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01992 · Cited by 302 · see citing works

Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation

Xia, Y., Yang, D., Yu, Z., Liu, F., Cai, J., Yu, L., Zhu, Z., Xu, D., Yuille, A., & Roth, H. R. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101766 · Cited by 280 · see citing works

Untitled work

(2011) · Cited by 277 · see citing works

Missing Modalities Imputation via Cascaded Residual Autoencoder

Tran, L., Liu, X., Zhou, J., & Jin, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.528 · Cited by 256 · see citing works

RUBi: Reducing Unimodal Biases in Visual Question AnsweringPreprint

Cadène, R., Dancette, C., Ben-younes, H., Cord, M., & Parikh, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10169 · Cited by 255 · see citing works

Video-based emotion recognition in the wild using deep transfer learning and score fusion

Kaya, H., Gürpınar, F., & Salah, A. A. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.012 · Cited by 235 · see citing works

Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis

Akhtar, M. S., Chauhan, D. S., Ghosal, D., Poria, S., Ekbal, A., & Bhattacharyya, P. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1034 · Cited by 233 · see citing works

A Shared Task on Multimodal Machine Translation and Crosslingual Image Description

Specia, L., Frank, S., Sima’an, K., & Elliott, D. (2016) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w16-2346 · Cited by 228 · see citing works

Weakly-Shared Deep Transfer Networks for Heterogeneous-Domain Knowledge Propagation

Shu, X., Qi, G.-J., Tang, J., & Wang, J. (2015) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2733373.2806216 · Cited by 218 · see citing works

Towards Automatic Learning of Procedures from Web Instructional VideosPreprint

Zhou, L., Xu, C., & Corso, J. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.09788 · Cited by 216 · see citing works

A Review on Explainability in Multimodal Deep Neural Nets

Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works

Modality Distillation with Multiple Stream Networks for Action Recognition

Garcia, N. C., Morerio, P., & Murino, V. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_7 · Cited by 201 · see citing works

Decoupling "when to update" from "how to update"

Malach, E., & Shalev‐Shwartz, S. (2017) · neural information processing systems · Cited by 201 · see citing works

MFAS: Multimodal Fusion Architecture Search

Perez-Rua, J.-M., Vielzeuf, V., Pateux, S., Baccouche, M., & Jurie, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00713 · Cited by 193 · see citing works

Learning Alignment for Multimodal Emotion Recognition from Speech

Xu, H., Zhang, H., Han, K., Wang, Y., Peng, Y., & Li, X. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3247 · Cited by 185 · see citing works

A Simple Way to Make Neural Networks Robust Against Diverse Image Corruptions

Rusak, E., Schott, L., Zimmermann, R. S., Bitterwolf, J., Bringmann, O., Bethge, M., & Brendel, W. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_4 · Cited by 182 · see citing works

CentralNet: A Multilayer Approach for Multimodal Fusion

Vielzeuf, V., Lechervy, A., Pateux, S., & Jurie, F. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11024-6_44 · Cited by 166 · see citing works

Review Study of Interpretation Methods for Future Interpretable Machine Learning

Mi, J., Li, A., & Zhou, L. (2020) · IEEE Access · doi:10.1109/access.2020.3032756 · Cited by 162 · see citing works

GRAM: Graph-based Attention Model for Healthcare Representation LearningPreprint

Choi, E., Bahadori, M. T., Song, L., Stewart, W. F., & Sun, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07012 · Cited by 158 · see citing works

Probing the Need for Visual Context in Multimodal Machine Translation

Çağlayan, O., Madhyastha, P., Specia, L., & Barrault, L. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1422 · Cited by 149 · see citing works

Self-training Improves Pre-training for Natural Language Understanding

Du, J., Grave, É., Gunel, B., Chaudhary, V., Çelebi, O., Auli, M., Stoyanov, V., & Conneau, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.426 · Cited by 141 · see citing works

A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis

Delbrouck, J.-B., Tits, N., Brousmiche, M., & Dupont, S. (2020) · doi:10.18653/v1/2020.challengehml-1.1 · Cited by 139 · see citing works

Multiple Classifier Systems for the Classification of Audio-Visual Emotional States

Glodek, M., Tschechne, S., Layher, G., Schels, M., Brosch, T., Scherer, S. J., Kächele, M., Schmidt, M., Heiko Neumann, B., Palm, G., et al. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_47 · Cited by 133 · see citing works

FairFace: Face Attribute Dataset for Balanced Race, Gender, and AgePreprint

Kärkkäinen, K., & Joo, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04913 · Cited by 129 · see citing works

Heterogeneous Domain Generalization Via Domain Mixup

Wang, Y., Li, H., & Kot, A. C. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053273 · Cited by 122 · see citing works

Affective Computing and Intelligent Interaction

D’Mello, S., Martin, J.-C., Schuller, B., & Graesser, A. C. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8 · Cited by 122 · see citing works

Contrastive Learning for Weakly Supervised Phrase Grounding

Gupta, T., Vahdat, A., Chechik, G., Yang, X., Kautz, J., & Hoiem, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_44 · Cited by 121 · see citing works

Gas Detection and Identification Using Multimodal Artificial Intelligence Based Sensor Fusion

Narkhede, P., Walambe, R. A., Mandaokar, S., Chandel, P., Kotecha, K. V., & Ghinea, G. (2021) · Applied System Innovation · doi:10.3390/asi4010003 · Cited by 121 · see citing works

Decoupling "when to update" from "how to update"Preprint

Malach, E., & Shalev‐Shwartz, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02613 · Cited by 118 · see citing works

Multimodal Prototypical Networks for Few-shot Learning

Pahde, F., Puscas, M., Klein, T., & Nabi, M. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00269 · Cited by 116 · see citing works

Towards Unsupervised Image Captioning With Shared Multimodal Embeddings

Laina, I., Rupprecht, C., & Navab, N. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00751 · Cited by 111 · see citing works

Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces

Gómez, L., Patel, Y., Rusiñol, M., Karatzas, D., & Jawahar, C. V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.218 · Cited by 106 · see citing works

Multimodal Learning with Incomplete Modalities by Knowledge Distillation

Wang, Q., Zhan, L., Thompson, P. M., & Zhou, J. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403234 · Cited by 102 · see citing works

Deep Multimodal Transfer Learning for Cross-Modal Retrieval

Zhen, L., Hu, P., Peng, X., Goh, R. S. M., & Zhou, J. T. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3029181 · Cited by 94 · see citing works

CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation

Hao, W., Zhang, Z., & Guan, H. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12329 · Cited by 81 · see citing works

A Comprehensive Introduction to Label Noise

Frénay‬, B., & Kabán, A. (2014) · Cited by 79 · see citing works

Cross-Subject and Cross-Modal Transfer for Generalized Abnormal Gait Pattern Recognition

Gu, X., Guo, Y., Deligianni, F., Lo, B. P. L., & Yang, G. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3009448 · Cited by 73 · see citing works

Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attention

Duan, B., Tang, H., Wang, W., Zong, Z., Yang, G., & Yan, Y. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00406 · Cited by 70 · see citing works

Multimodal Disentangled Domain Adaption for Social Media Event Rumor Detection

Zhang, H., Qian, S., Fang, Q., & Xu, C. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3042055 · Cited by 69 · see citing works

Missing Modality Transfer Learning via Latent Low-Rank Constraint

Ding, Z., Shao, M., & Fu, Y. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2462023 · Cited by 68 · see citing works

Label-Noise Robust Generative Adversarial Networks

Kaneko, T., Ushiku, Y., & Harada, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00257 · Cited by 66 · see citing works

Clustering on Multiple Incomplete Datasets via Collective Kernel Learning

Shao, W., Shi, X., & Yu, P. S. (2013) · Proceedings · doi:10.1109/icdm.2013.117 · Cited by 63 · see citing works

Deep multisensor learning for missing-modality all-weather mapping

Zheng, Z., Ma, A., Zhang, L., & Zhong, Y. (2021) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2020.12.009 · Cited by 63 · see citing works

A Unified Framework for Multimodal Domain Adaptation

Qi, F., Yang, X., & Xu, C. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240633 · Cited by 62 · see citing works

Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models

Huang, P.-Y., Patrick, M., Hu, J., Neubig, G., Metze, F., & Hauptmann, A. G. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.195 · Cited by 62 · see citing works

MasakhaNER: Named entity recognition for African languages

(2021) · HAL · doi:10.1162/tacl · Cited by 61 · see citing works

Deep Manifold-to-Manifold Transforming Network for Skeleton-based Action Recognition

Zhang, T., Zheng, W., Cui, Z., Zong, Y., Li, C., Zhou, X., & Yang, J. J. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2966878 · Cited by 59 · see citing works

A cross-disciplinary comparison of multimodal data fusion approaches and applications: Accelerating learning through trans-disciplinary information sharing

Bokade, R., Navato, A. P., Ouyang, R., Jin, X., Chou, C.-A., Ostadabbas, S., & Mueller, A. V. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113885 · Cited by 59 · see citing works

CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation LearningPreprint

Peng, Y., Qi, J., & Yuan, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.05106 · Cited by 58 · see citing works

Multimodal Encoder-Decoder Attention Networks for Visual Question Answering

Chen, C., Han, D., & Wang, J. (2020) · IEEE Access · doi:10.1109/access.2020.2975093 · Cited by 58 · see citing works

Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning

Rajendran, J., Khapra, M. M., Chandar, S., & Ravindran, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1021 · Cited by 57 · see citing works

Cross-Modal Data Programming Enables Rapid Medical Machine Learning

Dunnmon, J., Ratner, A., Saab, K. K., Khandwala, N., Markert, M. S., Sagreiya, H., Goldman, R. E., Lee‐Messer, C., Lungren, M. P., Rubin, D. L., et al. (2020) · Patterns · doi:10.1016/j.patter.2020.100019 · Cited by 57 · see citing works

Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think!

Hessel, J., & Lee, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.62 · Cited by 57 · see citing works

Multi-Task Learning of Hierarchical Vision-Language Representation

Nguyen, D.-K., & Okatani, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01074 · Cited by 55 · see citing works

Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zero-shot Classification and Retrieval of Videos

Parida, K. K., Matiyali, N., Guha, T., & Sharma, G. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093438 · Cited by 49 · see citing works

Multiple Instance Learning

Herrera, F., Ventura, S., Bello, R., Chris Cornelis, A., Zafra, A., Sánchez-Tarragó, D., & Vluymans, S. (2016) · Springer eBooks · doi:10.1007/978-3-319-47759-6_2 · Cited by 42 · see citing works

Complex Object Classification

Yang, Y., Wu, Y.-F., Zhan, D., Liu, Z., & Jiang, Y. Y. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3220012 · Cited by 41 · see citing works

Self-training Improves Pre-training for Natural Language UnderstandingPreprint

Du, J., Grave, É., Gunel, B., Chaudhary, V., Çelebi, O., Auli, M., Stoyanov, V., & Conneau, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02194 · Cited by 41 · see citing works

Cross-Dataset Adaptation for Visual Question Answering

Sha, F., Hu, H., & Chao, W. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00599 · Cited by 40 · see citing works

Foundations of Multimodal Co-learning

Zadeh, A., Liang, P. P., & Morency, L.-P. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.001 · Cited by 40 · see citing works

WordNet: An electronic lexical database. Christiane Fellbaum (Ed.). Cambridge, MA: MIT Press, 1998. Pp. 423.

Oram, P. (2001) · Applied Psycholinguistics · doi:10.1017/s0142716401221079 · Cited by 38 · see citing works

Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesPreprint

Pham, H., Liang, P. P., Manzini, T., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.07809 · Cited by 38 · see citing works

Robust Student Network Learning

Guo, T., Xu, C., He, S., Shi, B., Xu, C., & Tao, D. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2929114 · Cited by 38 · see citing works

HMTL: Heterogeneous Modality Transfer Learning for Audio-Visual Sentiment Analysis

Seo, S., Na, S., & Kim, J. (2020) · IEEE Access · doi:10.1109/access.2020.3006563 · Cited by 38 · see citing works

Audio–visual domain adaptation using conditional semi-supervised Generative Adversarial Networks

Athanasiadis, C., Hortal, E., & Asteriadis, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.09.106 · Cited by 37 · see citing works

MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding

Wang, Q., Tan, H. X., Shen, S., Mahoney, M. W., & Yao, Z. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.159 · Cited by 37 · see citing works

Multiple Instance Learning: Foundations and Algorithms

Herrera, F., Ventura, S., Bello, R., Chris Cornelis, A., Zafra, A., Snchez-Tarrag, D., & Vluymans, S. (2016) · Springer eBooks · doi:10.1007/978-3-319-47759-6 · Cited by 36 · see citing works

A Closer Look at the Robustness of Vision-and-Language Pre-trained ModelsPreprint

Li, L., Gan, Z., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08673 · Cited by 31 · see citing works

Multimodal Human Computer Interaction and Pervasive Services

Grifoni, P. (2009) · IGI Global eBooks · doi:10.4018/978-1-60566-386-9 · Cited by 30 · see citing works

Multimodal Transfer Deep Learning with Applications in Audio-Visual RecognitionPreprint

Moon, S., Kim, S., & Wang, H. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.3121 · Cited by 29 · see citing works

Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data

Fayek, H. M., & Kumar, A. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/78 · Cited by 29 · see citing works

Deep Co-Attention Network for Multi-View Subspace Learning

Zheng, L., Cheng, Y., Yang, H., Cao, N., & He, J. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449801 · Cited by 28 · see citing works

Multimodal deep generative adversarial models for scalable doubly semi-supervised learning

Du, C., Du, C., & He, H. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.11.003 · Cited by 27 · see citing works

Associative Multichannel Autoencoder for Multimodal Word Representation

Wang, S., Zhang, J., & Zong, C. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1011 · Cited by 25 · see citing works

Modularized Textual Grounding for Counterfactual Resilience

Fang, Z., Kong, S., Fowlkes, C. C., & Yang, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00654 · Cited by 25 · see citing works

Counterfactual VQA: A Cause-Effect Look at Language BiasPreprint

Niu, Y., Tang, K., Zhang, H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04315 · Cited by 23 · see citing works

Enriched music representations with multiple cross-modal contrastive learning

Ferraro, A., Favory, X., Drossos, K., Kim, Y., & Bogdanov, D. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3071082 · Cited by 23 · see citing works

Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment

Liang, P. P., Wu, P., Ziyin, L., Morency, L.-P., & Salakhutdinov, R. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475247 · Cited by 22 · see citing works

Unsupervised Audio-Visual Subspace Alignment for High-Stakes Deception Detection

Mathur, L., & Matarić, M. J. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413550 · Cited by 22 · see citing works

Multimodal Aggregation Approach for Memory Vision-Voice Indoor Navigation with Meta-Learning

Yan, L., Liu, D., Song, Y., & Yu, C. B. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341398 · Cited by 22 · see citing works

Relating by Contrasting: A Data-efficient Framework for Multimodal Generative ModelsPreprint

Shi, Y., Paige, B., Torr, P. H. S., & Siddharth, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.01179 · Cited by 21 · see citing works

Intelligence Science and Big Data Engineering

Sun, C., Fang, F., Zhou, Z.-H., et al. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-42057-3 · Cited by 21 · see citing works

ViCo: Word Embeddings From Visual Co-Occurrences

Gupta, T., Schwing, A. G., & Hoiem, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00752 · Cited by 20 · see citing works

MT-GCN for Multi-label Audio-tagging with Noisy Labels

Shrivastava, H., Yin, Y., Shah, R. R., & Zimmermann, R. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053065 · Cited by 20 · see citing works

Computer Vision – ECCV 2020

Online, E. 2., Vedaldi, A., Bischof, H., Brox, T., & Frahm, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58545-7 · Cited by 20 · see citing works

Audio-Visual Speech Enhancement based on Multimodal Deep Convolutional Neural Network.Preprint

Hou, J.-C., Wang, S., Lai, Y.-H., Lin, J., Tsao, Y., Chang, H.-W., & Wang, H. (2017) · arXiv (Cornell University) · Cited by 19 · see citing works

Multimodal learning with transformers: A survey

Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works

Multi-fault diagnosis of Industrial Rotating Machines using Data-driven approach : A review of two decades of research

Gawde, S., Patil, S., Kumar, S., Kamat, P. V., Kotecha, K. V., & Abraham, A. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106139 · Cited by 327 · see citing works

Domain Adaptation: Challenges, Methods, Datasets, and Applications

Singhal, P., Walambe, R. A., Ramanna, S., & Kotecha, K. V. (2023) · IEEE Access · doi:10.1109/access.2023.3237025 · Cited by 253 · see citing works

A survey of multimodal hybrid deep learning for computer vision: Architectures, applications, trends, and challenges

Bayoudh, K. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102217 · Cited by 173 · see citing works

Review of the Li-Ion Battery, Thermal Management, and AI-Based Battery Management System for EV Application

Ghalkhani, M., & Habibi, S. R. (2022) · Energies · doi:10.3390/en16010185 · Cited by 151 · see citing works

A comprehensive investigation of multimodal deep learning fusion strategies for breast cancer classification

Nakach, F.-Z., Idri, A., & Göçeri, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10984-z · Cited by 143 · see citing works

A review on EEG-based multimodal learning for emotion recognition

Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 141 · see citing works

A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)

Deldjoo, Y., He, Z., McAuley, J., Korikov, A., Sanner, S., Arnau Ramisa, A., Vidal, R., Sathiamoorthy, M., Kasirzadeh, A., & Milano, S. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671474 · Cited by 141 · see citing works

Dynamic Vision-Based Machinery Fault Diagnosis with Cross-Modality Feature Alignment

Li, X., Yu, S., Lei, Y., Li, N., & Yang, B. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124470 · Cited by 112 · see citing works

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

Barua, A., Ahmed, M. U., & Begum, S. (2023) · IEEE Access · doi:10.1109/access.2023.3243854 · Cited by 109 · see citing works

From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape

McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works

Multimodal Alignment and Fusion: A Survey

Li, S., & Tang, H. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02667-1 · Cited by 191 · see citing works

Visuo-haptic object perception for robots: an overview

Navarro-Guerrero, N., Toprak, S., Josifovski, J., & Jamone, L. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10091-y · Cited by 78 · see citing works

GaitFi: Robust Device-Free Human Identification via WiFi and Vision Multimodal Learning

Deng, L., Yang, J., Yuan, S., Zou, H., Lu, C. X., & Xie, L. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3203559 · Cited by 75 · see citing works

Federated Learning on Multimodal Data: A Comprehensive Survey

Lin, Y., Gao, Y., Gong, M., Zhang, S., Zhang, Y., & Li, Z. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1398-0 · Cited by 73 · see citing works

An effective multimodal representation and fusion method for multimodal intent recognition

Huang, X., Ma, T., Jia, L., Zhang, Y., Rong, H., & Al-Nabhan, N. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126373 · Cited by 72 · see citing works

Latent information-guided one-step multi-view fuzzy clustering based on cross-view anchor graph

Zhang, C., Chen, L., Shi, Z., & Ding, W. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102025 · Cited by 72 · see citing works

Reviewing Multimodal Machine Learning and Its Use in Cardiovascular Diseases Detection

Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2023) · Electronics · doi:10.3390/electronics12071558 · Cited by 75 · see citing works

Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications

Manzoor, M. A., Albarri, S., Xian, Z., Meng, Z., Nakov, P., & Liang, S. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3617833 · Cited by 76 · see citing works

A novel multi-label pest image classifier using the modified Swin Transformer and soft binary cross entropy loss

Guo, Q., Wang, C., Xiao, D., & Huang, Q. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107060 · Cited by 68 · see citing works

Machine learning for cognitive behavioral analysis: datasets, methods, paradigms, and research directions

Bhatt, P. R., Sethi, A., Tasgaonkar, V., Shroff, J., Pendharkar, I., Desai, A., Sinha, P., Deshpande, A., Joshi, G., Rahate, A., et al. (2023) · Brain Informatics · doi:10.1186/s40708-023-00196-6 · Cited by 75 · see citing works

Rice Transformer: A Novel Integrated Management System for Controlling Rice Diseases

Patil, R. R., & Kumar, S. (2022) · IEEE Access · doi:10.1109/access.2022.3200688 · Cited by 59 · see citing works

Excavating multimodal correlation for representation learning

Mai, S., Ya, S., Zeng, Y., & Hu, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.003 · Cited by 57 · see citing works

Two Birds With One Stone: Knowledge-Embedded Temporal Convolutional Transformer for Depression Detection and Emotion Recognition

Zheng, W., Yan, L., & Wang, F. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3282704 · Cited by 55 · see citing works

A deep fusion model for stock market prediction with news headlines and time series data

Chen, P., Boukouvalas, Z., & Corizzo, R. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10303-1 · Cited by 51 · see citing works

Multimodal vision-based human action recognition using deep learning: a review

Shafizadegan, F., Naghsh‐Nilchi, A. R., & Shabaninia, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10730-5 · Cited by 55 · see citing works

Learning across Diverse Biomedical Data Modalities and Cohorts: Challenges and Opportunities for Innovation

Rajendran, S., Pan, W., Sabuncu, M. R., Chen, Y., Zhou, J., & Wang, F. (2024) · Patterns · doi:10.1016/j.patter.2023.100913 · Cited by 47 · see citing works

Automated residential layout generation and editing using natural language and images

Zeng, P., Gao, W., Li, J., Yin, J., Chen, J., & Lu, S. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106133 · Cited by 45 · see citing works

Deep learning methods improve genomic prediction of wheat breeding

Montesinos‐López, A., Crespo‐Herrera, L., Dreisigacker, S., Gerard, G., Vitale, P., Pierre, C. S., Govindan, V., Tadesse, Z., Flores, M. C., Pérez‐Rodríguez, P., et al. (2024) · Frontiers in Plant Science · doi:10.3389/fpls.2024.1324090 · Cited by 43 · see citing works

Incorporating stock prices and text for stock movement prediction based on information fusion

Zhang, Q., Zhang, Y., Bao, F., Liu, Y., Zhang, C., & Лю, П. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107377 · Cited by 43 · see citing works

Employing multimodal co-learning to evaluate the robustness of sensor fusion for industry 5.0 tasks

Rahate, A., Mandaokar, S., Chandel, P., Walambe, R. A., Ramanna, S., & Kotecha, K. V. (2022) · Soft Computing · doi:10.1007/s00500-022-06802-9 · Cited by 39 · see citing works

Literature survey of multi-track music generation model based on generative confrontation network in intelligent composition

Liu, W. (2022) · The Journal of Supercomputing · doi:10.1007/s11227-022-04914-5 · Cited by 39 · see citing works

Clinical applications of graph neural networks in computational histopathology: A review

Meng, X., & Zou, T. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107201 · Cited by 38 · see citing works

Multimodal Colearning Meets Remote Sensing: Taxonomy, State of the Art, and Future Works

Kieu, N., Nguyen, K., Nazib, A., Fernando, T., Fookes, C., & Sridharan, S. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3378348 · Cited by 38 · see citing works

Multimodal fusion techniques: Review, data representation, information fusion, and application areas

Hangloo, S., & Arora, B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130827 · Cited by 43 · see citing works

Adapt and explore: Multimodal mixup for representation learning

Lin, R., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102216 · Cited by 34 · see citing works

Survey of Multimodal Federated Learning: Exploring Data Integration, Challenges, and Future Directions

Adam, M., Albaseer, A. M., Baroudi, U. A., & Abdallah, M. M. (2025) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2025.3554537 · Cited by 36 · see citing works

Personalized uncertainty quantification in artificial intelligence

Chakraborti, T., Banerji, C. R. S., Marandon, A., Hellon, V., Mitra, R., Lehmann, B., Bräuninger, L., McGough, S. F., Turkay, Ç., Frangi, A. F., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01024-8 · Cited by 37 · see citing works

A survey on advancements in image–text multimodal models: From general techniques to biomedical implementations

Guo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 50 · see citing works

Backdooring Multimodal Learning

Han, X., Wu, Y., Zhang, Q., Zhou, Y. H., Xu, Y., Qiu, H., Xu, G., & Zhang, T. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00031 · Cited by 33 · see citing works

Physical knowledge-driven feature fusion and reconstruction network for fault diagnosis with incomplete multisource data

Sun, D., Li, Y., Jia, S., Gao, S., Noman, K., & Eliker, K. (2024) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2024.112222 · Cited by 32 · see citing works

Multimodal Masked Autoencoder Based on Adaptive Masking for Vitiligo Stage Classification

Fan, X., Li, Z., Jiang, S. S., Li, C., Li, S., Gao, T., He, K., Chen, J., Zhang, J., & Zhang, J. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01521-7 · Cited by 32 · see citing works

Missing-modality enabled multi-modal fusion architecture for medical data

Wang, M., Fan, S., Li, Y., Xie, Z., & Chen, H. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104796 · Cited by 31 · see citing works

Operating Condition Recognition of Industrial Flotation Processes Using Visual and Acoustic Bimodal Autoencoder With Manifold Learning

Liu, C., Wang, Y., Fang, Y., Yang, C., & Gui, W. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3359416 · Cited by 30 · see citing works

Energy consumption intelligent modeling and prediction for additive manufacturing via multisource fusion and inter-layer consistency

Wang, K., Yu, L., Xu, J., Zhang, S., & Qin, J. (2022) · Computers & Industrial Engineering · doi:10.1016/j.cie.2022.108720 · Cited by 31 · see citing works

Multimodal Boosting: Addressing Noisy Modalities and Identifying Modality Contribution

Mai, S., Ya, S., Xiong, A., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3306489 · Cited by 29 · see citing works

Energy efficiency design for eco-friendly additive manufacturing based on multimodal attention fusion

Wang, K., Song, Y., Sheng, H., Xu, J., Zhang, S., & Qin, J. (2022) · Journal of Manufacturing Processes · doi:10.1016/j.jmapro.2022.05.010 · Cited by 29 · see citing works

Multi-Modal CNN Features Fusion for Emotion Recognition: A Modified Xception Model

Shahzad, H. M., Bhatti, S. M., Jaffar, A., Rashid, M., & Akram, S. (2023) · IEEE Access · doi:10.1109/access.2023.3310428 · Cited by 28 · see citing works

Multimodal Deep Learning Integration of Image, Weather, and Phenotypic Data Under Temporal Effects for Early Prediction of Maize Yield

Shamsuddin, D., Danilevicz, M. F., Al-Mamun, H. A., Bennamoun, M., & Edwards, D. (2024) · Remote Sensing · doi:10.3390/rs16214043 · Cited by 29 · see citing works

A review and critical analysis of multimodal datasets for emotional AI

Al-Azani, S., & El-Alfy, E.-S. M. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11271-1 · Cited by 32 · see citing works

A co-learning method to utilize optical images and photogrammetric point clouds for building extraction

Xie, Y., Tian, J., & Zhu, X. X. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.103165 · Cited by 27 · see citing works

Multimodal unified generalization and translation network for intelligent fault diagnosis under dynamic environments

Zhao, C., Shen, W., Zio, E., & Ma, H. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112559 · Cited by 26 · see citing works

Additive manufacturing energy consumption measurement and prediction in fabricating lattice structure based on recallable multimodal fusion network

Wang, K., Song, Y., Huang, Z., Sun, Y., Xu, J., & Zhang, S. (2022) · Measurement · doi:10.1016/j.measurement.2022.111215 · Cited by 26 · see citing works

Deep learning on multi-view sequential data: a survey

Xie, Z., Yang, Y. Z., Zhang, Y., Wang, J., & Du, S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10332-z · Cited by 28 · see citing works

Alignment-Enhanced Interactive Fusion Model for Complete and Incomplete Multimodal Hand Gesture Recognition

Duan, S., Wu, L., Liu, A., & Chen, X. (2023) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2023.3335101 · Cited by 25 · see citing works

Intent recognition model based on sequential information and sentence features

Wu, T., Wang, M., Xi, Y., & Zhao, Z. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127054 · Cited by 24 · see citing works

Multi-modal learning-based algae phyla identification using image and particle modalities

Kwon, D. H., Lee, M. J., Jeong, H., Park, S., & Cho, K. H. (2025) · Water Research · doi:10.1016/j.watres.2025.123172 · Cited by 24 · see citing works

Building Multimodal Knowledge Bases With Multimodal Computational Sequences and Generative Adversarial Networks

Chen, D., & Zhang, R. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3291503 · Cited by 23 · see citing works

A Short Survey on Deep Learning for Multimodal Integration: Applications, Future Perspectives and Challenges

Dimitri, G. M. (2022) · Computers · doi:10.3390/computers11110163 · Cited by 22 · see citing works

Research on the perception of the terrain image of the tourism destination based on multimodal user-generated content data

Hu, T., & Geng, J. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1801 · Cited by 21 · see citing works

A Review of the Applications, Benefits, and Challenges of Generative AI for Sustainable Toxicology

Alam, F., Alnazzawi, T. S. M., Mehmood, R., & Almaghthawi, A. (2025) · Current Research in Toxicology · doi:10.1016/j.crtox.2025.100232 · Cited by 23 · see citing works

Navigating the Multimodal Landscape: A Review on Integration of Text and Image Data in Machine Learning Architectures

Rashid, M. B., Rahaman, M. S., & Rivas, P. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6030074 · Cited by 23 · see citing works

TS2ACT: Few-Shot Human Activity Sensing with Cross-Modal Co-Learning

Xia, K., Li, W., Gan, S., & Lu, S. (2023) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3631445 · Cited by 19 · see citing works

Multimedia analysis of robustly optimized multimodal transformer based on vision and language co-learning

Yoon, J., Choi, G. H., & Choi, C. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101922 · Cited by 19 · see citing works

Deep Learning models for cultural pattern recognition: preserving intangible heritage of Li ethnic subgroups through intelligent documentation systems

Sun, J., Khalid, K. A. T., & Kay, C. S. (2025) · Future Technology · doi:10.55670/fpll.futech.4.3.12 · Cited by 18 · see citing works

Multimodal representation learning over heterogeneous networks for tag-based music retrieval

da Silva, A. C. M., Silva, D. F., & Marcacini, R. M. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117969 · Cited by 17 · see citing works

Employing Co-Learning to Evaluate the Explainability of Multimodal Sentiment Analysis

Jain, D. K., Rahate, A., Joshi, G., Walambe, R. A., & Kotecha, K. V. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3176403 · Cited by 17 · see citing works

Cross-Modal Audio-Visual Co-Learning for Text-Independent Speaker Verification

Liu, M., Lee, K. A., Wang, L., Zhang, H., Zeng, C., & Dang, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095883 · Cited by 17 · see citing works

Attention-Based Multimodal Deep Learning on Vision-Language Data: Models, Datasets, Tasks, Evaluation Metrics and Applications

Bose, P., Rana, P., & Ghosh, P. (2023) · IEEE Access · doi:10.1109/access.2023.3299877 · Cited by 16 · see citing works

Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learning

Lai, S., Li, J., Guo, G., Hu, X., Li, Y., Tan, Y., Song, Z., Liu, Y., Ren, Z., Wang, C., et al. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651442 · Cited by 16 · see citing works

TransG-net: transformer and graph neural network based multi-modal data fusion network for molecular properties prediction

Zhang, T., Chen, S., Wulamu, A., Guo, X., Li, Q., & Zheng, H. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04351-0 · Cited by 16 · see citing works

An Empirical Study of Adaptive Feedback to Enhance Cognitive Ability in Programming Learning among College Students: A Perspective Based on Multimodal Data Analysis

Fu, W.-S., Zhang, J., Zhang, D., Li, T., Lan, M., & Liu, N. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331241313126 · Cited by 15 · see citing works

Emotion Recognition by Learning the Manifold of Fused Multiscale Information of EEG Signals

Li, C., Zhang, S., Mu, Y., Yang, L., Peng, Y., Li, F., Zhang, Y., Liang, Z., Cao, Z., Wan, F., et al. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3555226 · Cited by 15 · see citing works

A Sign Language Recognition Framework Based on Cross-Modal Complementary Information Fusion

Zhang, J., Wang, Q., & Wang, Q. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3377095 · Cited by 14 · see citing works

Make it till you fake it: Construction-centric computational framework for simultaneous image synthetization and multimodal labeling

Tohidifar, A., Kim, D., & Lee, S. H. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105696 · Cited by 14 · see citing works

Learning incremental audio–visual representation for continual multimodal understanding

Zhu, B., Wang, C., Xu, K., Feng, D., Zhou, Z., & Zhu, X. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112513 · Cited by 14 · see citing works

Coco-Attention for Tumor Segmentation in Weakly Paired Multimodal MRI Images

Zhou, P., Li, Y., Chen, H., & Peng, Y. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3262548 · Cited by 14 · see citing works

Alignment-assisted Frequency Fusion Network for RGB-infrared vehicle detection

Chen, Z., Xiang, W., Lin, Z., Yang, K., Liu, Y., & Shi, Z. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130505 · Cited by 13 · see citing works

Multimodal large language models for medical image diagnosis: Challenges and opportunities

Zhang, A., Zhao, E., Wang, R., Zhang, X., Wang, J., & Chen, E. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104895 · Cited by 14 · see citing works

Intrinsic explainability of multimodal learning for crop yield simulation

Najjar, H., Pathak, D., Nuske, M., & Dengel, A. R. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.111003 · Cited by 13 · see citing works

Multimodal learning for fetal distress diagnosis using a multimodal medical information fusion framework

Zhang, Y., Deng, Y., Zhou, Z., Zhang, X., Jiao, P., & Zhao, Z. (2022) · Frontiers in Physiology · doi:10.3389/fphys.2022.1021400 · Cited by 13 · see citing works

A Collaborative Multimodal Learning-Based Framework for COVID-19 Diagnosis

Gao, Y., Gong, M., Ong, Y.-S., Qin, A. K., Wu, Y., & Xie, F. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3290188 · Cited by 13 · see citing works

An fNIRS representation and fNIRS-scales multimodal fusion method for auxiliary diagnosis of amnestic mild cognitive impairment

Cheng, S., Shang, P., Zhang, Y., Guan, J., Chen, Y., Lv, Z., Huang, S., Liu, Y., & Xie, H. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106646 · Cited by 13 · see citing works

SoK: Behind the Accuracy of Complex Human Activity Recognition Using Deep Learning

Nguyen, D., & Le‐Khac, N. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650322 · Cited by 13 · see citing works

Investigation and Empirical Analysis of Transfer Learning for Industrial IoT Networks

Yadav, P., Rishiwal, V., Yadav, M., Alotaibi, A., Maurya, V., Agarwal, U., & Sharma, S. (2024) · IEEE Access · doi:10.1109/access.2024.3499741 · Cited by 13 · see citing works

Relative Norm Alignment for Tackling Domain Shift in Deep Multi-modal Classification

Planamente, M., Plizzari, C., Peirone, S. A., Barbara Caputo, B., & Bottino, A. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-01998-9 · Cited by 12 · see citing works

Multimodal-Based Learning Improves Students' Reading Ability in Perspective of Systemic Functional Linguistics

Purba, R., Herman, H., Fatmawati, E., Saputra, N., & Sabata, Y. N. (2023) · Journal of Education Technology · doi:10.23887/jet.v7i1.54336 · Cited by 12 · see citing works

Multimodal deep learning for allergenic proteins prediction

Yu, L., Luo, Y., Wu, S., Chen, S., Xue, L., Jing, R., & Luo, J. (2025) · BMC Biology · doi:10.1186/s12915-025-02347-z · Cited by 11 · see citing works

Mining and reasoning of data uncertainty-induced imprecision in deep image classification

Zhang, Z., Ning, L., Liu, Z., Yang, Q., & Ding, W. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.03.014 · Cited by 11 · see citing works

Multimodal Co-Learning for Building Change Detection: A Domain Adaptation Framework Using VHR Images and Digital Surface Models

Xie, Y., Yuan, X., Zhu, X. X., & Tian, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3362680 · Cited by 11 · see citing works

A Social Media Dataset and H-GNN-Based Contrastive Learning Scheme for Multimodal Sentiment Analysis

Peng, J., He, Y., Chang, Y., Lu, Y., Zhang, P., Ou, Z., & Yu, Q. (2025) · Applied Sciences · doi:10.3390/app15020636 · Cited by 11 · see citing works

A Stack Based Multimodal Machine Learning Model for Breast Cancer Diagnosis

Kayıkçı, Ş., & Khoshgoftaar, T. M. (2022) · International Congress on Human-Computer Interaction, Optimization and Robotic Applications (ICHORA) · doi:10.1109/hora55278.2022.9800004 · Cited by 11 · see citing works

Styled and characteristic Peking opera facial makeup synthesis with co-training and transfer conditional styleGAN2

Shen, Y., Duan, O., Xin, X., Yan, M. G., & Li, Z. (2024) · Heritage Science · doi:10.1186/s40494-024-01463-3 · Cited by 9 · see citing works

Transfer Learning of Human Activities Based on IMU Sensors: A Review

Ashry, S., Das, S., Rafiei, M., Baumbach, J., & Baumbach, L. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3510097 · Cited by 10 · see citing works

FDFR-Net: A fruit ripeness classification method using multimodal learning technique

Wang, Y., Jin, X., Huang, W., Wang, X., & Yu, W. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130126 · Cited by 9 · see citing works

Evidence combination with multi-granularity belief structure for pattern classification

Zuo, K., Li, X., Yu, L., Shen, T., Dong, Y., & Dezert, J. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121577 · Cited by 8 · see citing works

Cross-language dissemination of Chinese classical literature using multimodal deep learning and artificial intelligence

Bai, Y., & Lei, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-05921-1 · Cited by 8 · see citing works

MDCNN: A multimodal dual-CNN recursive model for fake news detection via audio- and text-based speech emotion recognition

Wu, H., Li, H., Fang, X., Tang, M., Yu, H., Yu, B., Li, M., Jing, Z., Meng, Y., Chen, W., et al. (2025) · Speech Communication · doi:10.1016/j.specom.2025.103313 · Cited by 8 · see citing works

Comparison Analysis of Multimodal Fusion for Dangerous Action Recognition in Railway Construction Sites

Amel, O., Siebert, X., & Mahmoudi, S. A. (2024) · Electronics · doi:10.3390/electronics13122294 · Cited by 8 · see citing works

A survey on semiconductor wafer yield prediction by artificial intelligence

Zhao, D., Chen, Y., Guo, P., Chen, F., Jing, R., Xu, X., Chen, Y., Liu, F., & Wu, B. (2025) · Microelectronics Journal · doi:10.1016/j.mejo.2025.106958 · Cited by 7 · see citing works

Multimodal Learning and its Applications

Haddag, A. (2023) · doi:10.13140/rg.2.2.33217.57440

Tutorial on Multimodal Machine Learning: Principles, Challenges, and Open Questions

Liang, P. P., & Morency, L. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3610661.3617602 · Cited by 48 · see citing works

Deep Multimodal Learning with Missing Modality: A SurveyPreprint

Wu, R., Hu, W., Chen, H., & Carneiro, G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.07825 · Cited by 8 · see citing works

Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications

Manzoor, M. A., Albarri, S., Xian, Z., Meng, Z., Nakov, P., & Liang, S. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3617833 · Cited by 76 · see citing works

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

Barua, A., Ahmed, M. U., & Begum, S. (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7736385