A Unimodal Representation Learning and Recurrent Decomposition Fusion Structure for Utterance-Level Multimodal Embedding Learning
Abstract
Learning a unified embedding for utterance-level video attracts significant attention recently due to the rapid development of social media and its broad applications. An utterance normally contains not only spoken language but also the nonverbal behaviors such as facial expressions and vocal patterns. Instead of directly learning utterance embedding based on low-level features, we firstly explore high-level representation for each modality separately via an unimodal representation learning gyroscope structure. In this way, the learnt unimodal representations are more representative and contain more abstract semantic information. In the gyroscope structure, we introduce multi-scale kernel learning, ‘channel expansion’ and ‘channel fusion’ operations to explore high-level features both spatially and channelwise. Another insight of our method lies in that we fuse representations of all modalities to obtain a unified embedding by interpreting fusion procedure as the flow of inter-modality information between various modalities, which is more specialized in terms of the information to be fused and the fusion process. Specifically, considering that each modality carries modality-specific and cross-modality interactions, we innovate to decompose unimodal representations into intra- and inter-modality dynamics using gating mechanism, and further fuse the inter-modality dynamics by passing them from previous modalities to the following one using a recurrent neural fusion architecture. Extensive experiments demonstrate that our method achieves state-of-the-art performance on multiple benchmark datasets.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · see citing works
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,225 · see citing works
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513 · see citing works
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · see citing works
Convergence Results for Neural Networks via Electrodynamics
Panigrahy, R., Rahimi, A., Sachdeva, S., et al. (2018) · Leibniz international proceedings in informatics · doi:10.4230/lipics.itcs.2018.22 · Cited by 2,848 · see citing works
A Critical Review of Recurrent Neural Networks for Sequence LearningPreprint
Lipton, Z. C., Berkowitz, J. A., & Elkan, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00019 · Cited by 2,711 · see citing works
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · see citing works
Convolutional Neural Networks for Speech Recognition
Abdel‐Hamid, O., Mohamed, A., Jiang, H., Deng, L., Penn, G., & Yu, D. (2014) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2014.2339736 · Cited by 2,268 · see citing works
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259 · see citing works
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · see citing works
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · see citing works
MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., & Mihalcea, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1050 · Cited by 1,416 · see citing works
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170 · see citing works
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · doi:10.21236/ada623249 · Cited by 1,081 · see citing works
From Utterance to Text: The Bias of Language in Speech and Writing
Olson, D. (1977) · Harvard Educational Review · doi:10.17763/haer.47.3.8840364413869005 · Cited by 1,065 · see citing works
Context-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939 · see citing works
Memory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855 · see citing works
Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852 · see citing works
COVAREP — A collaborative voice analysis repository for speech technologies
Degottex, G., Kane, J., Drugman, T., Raitio, T., & Scherer, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853739 · Cited by 695 · see citing works
Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis
Poria, S., Chaturvedi, I., Cambria, E., & Hussain, A. (2016) · Proceedings · doi:10.1109/icdm.2016.0055 · Cited by 624 · see citing works
Speaker identification on the SCOTUS corpus
Yuan, J., & Liberman, M. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.2935783 · Cited by 600 · see citing works
VQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570 · see citing works
Semantics-preserving hashing for cross-view retrieval
Lin, Z., Ding, G., Hu, M., & Wang, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299011 · Cited by 563 · see citing works
ESPNetv2: A Light-Weight, Power Efficient, and General Purpose Convolutional Neural Network
Mehta, S., Rastegari, M., Shapiro, L., & Hajishirzi, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00941 · Cited by 551 · see citing works
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535 · see citing works
Multi-attention Recurrent Network for Human Communication Comprehension
Zadeh, A., Liang, P. P., Poria, S., Vij, P., Cambria, E., & Morency, L. (2018) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v32i1.12024 · Cited by 535 · see citing works
Pham, H., Liang, P. P., Manzini, T., Morency, L.-P., & Póczos, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016892 · Cited by 512 · see citing works
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Wang, Y., Shen, Y., Liu, Z., Liang, P. P., Zadeh, A., & Morency, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017216 · Cited by 493 · see citing works
Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos
Hazarika, D., Poria, S., Zadeh, A., Cambria, E., Morency, L., & Zimmermann, R. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1193 · Cited by 491 · see citing works
Towards multimodal sentiment analysis: harvesting opinions from the web
Morency, L.-P., Mihalcea, R., & Doshi, P. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070509 · Cited by 470 · see citing works
Youtube movie reviews: Sentiment analysis in an audio-visual context
Wollmer, M., Weninger, F., Knaup, T., Schuller, B., Sun, C., Sagae, K., & Morency, L.-P. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.34 · Cited by 403 · see citing works
Multimodal sentiment analysis using hierarchical fusion with context modeling
Majumder, N., Hazarika, D., Gelbukh, A. F., Cambria, E., & Poria, S. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.07.041 · Cited by 385 · see citing works
Multimodal sentiment analysis with word-level fusion and reinforcement learning
Chen, M., Wang, S., Liang, P. P., Baltrušaitis, T., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136801 · Cited by 334 · see citing works
Learning Factorized Multimodal Representations
Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., & Salakhutdinov, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06176 · Cited by 323 · see citing works
Wu, C., & Liang, W.-B. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.16 · Cited by 265 · see citing works
Deep multimodal fusion for persuasiveness prediction
Nojavanasghari, B., Gopinath, D., Koushik, J., Baltrušaitis, T., & Morency, L. (2016) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2993148.2993176 · Cited by 247 · see citing works
Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)
Castro, S., Hazarika, D., Pérez-Rosas, V., Zimmermann, R., Mihalcea, R., & Poria, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1455 · Cited by 243 · see citing works
Multimodal Language Analysis with Recurrent Multistage Fusion
Liang, P. P., Liu, Z., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1014 · Cited by 230 · see citing works
Multimodal Sentiment Analysis of Spanish Online Videos
Rosas, V. P., Mihalcea, R. F., & Morency, L. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.9 · Cited by 228 · see citing works
Multi-level Multiple Attentions for Contextual Multimodal Sentiment Analysis
Poria, S., Cambria, E., Hazarika, D., Mazumder, N., Zadeh, A., & Morency, L. (2017) · Proceedings · doi:10.1109/icdm.2017.134 · Cited by 215 · see citing works
UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
Hasan, M. K., Rahman, W., Bagher Zadeh, A., Zhong, J., Tanveer, M. I., Morency, L.-P., & Hoque, M. (. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1211 · Cited by 174 · see citing works
Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment
Gu, Y., Yang, K., Fu, S., Chen, S., Li, X., & Marsic, I. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1207 · Cited by 160 · see citing works
Deep Multimodal Learning for Affective Analysis and Retrieval
Pang, L., Zhu, S., & Ngo, C. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2482228 · Cited by 154 · see citing works
Mai, S., Hu, H., & Xing, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1046 · Cited by 153 · see citing works
Self-citationExtending Long Short-Term Memory for Multi-View Structured Learning
Rajagopalan, S. S., Morency, L., Baltrušaitis, T., & Goecke, R. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_21 · Cited by 138 · see citing works
Neural Machine Translation with Deep Attention
Zhang, B., Xiong, D., & Su, J. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2876404 · Cited by 114 · see citing works
Ensemble of SVM trees for multimodal emotion recognition
Rozgić, V., Ananthakrishnan, S., Saleem, S., Kumar, R., & Prasad, R. (2012) · Cited by 112 · see citing works
Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence
Zadeh, A., Chan, M., Liang, P. P., Tong, E., & Morency, L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00901 · Cited by 98 · see citing works
Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Liang, P. P., Liu, Z., Tsai, Y.-H. H., Zhao, Q., Salakhutdinov, R., & Morency, L.-P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1152 · Cited by 95 · see citing works
First-order versus second-order single-layer recurrent neural networks
Goudreau, M. W., Giles, C. L., Chakradhar, S., & Chen, D. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.286928 · Cited by 91 · see citing works
Mai, S., Xing, S., & Hu, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2925966 · Cited by 89 · see citing works
Self-citationMai, S., Xing, S., & Hu, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3068598 · Cited by 89 · see citing works
Self-citation(2019) · neural information processing systems · Cited by 83 · see citing works
Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
Pham, H., Manzini, T., Liang, P. P., & Póczos, B. (2018) · doi:10.18653/v1/w18-3308 · Cited by 76 · see citing works
Multi-Fusion Residual Memory Network for Multimodal Human Sentiment Comprehension
Mai, S., Hu, H., Jia, X., & Xing, S. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3000510 · Cited by 66 · see citing works
Self-citationMultimodal Local-Global Ranking Fusion for Emotion Recognition
Liang, P. P., Zadeh, A., & Morency, L. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3243019 · Cited by 45 · see citing works
Attentive matching network for few-shot learning
Mai, S., Hu, H., & Jia, X. (2019) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2019.07.001 · Cited by 43 · see citing works
Self-citationHuman Conversation Analysis Using Attentive Multimodal Networks with Hierarchical Encoder-Decoder
Gu, Y., Li, X., Huang, K., Fu, S., Yang, K., Chen, S., Zhou, M., & Marsic, I. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240714 · Cited by 39 · see citing works
Modality-based Factorization for Multimodal Fusion
Barezi, E. J., & Fung, P. (2019) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · doi:10.18653/v1/w19-4331 · Cited by 35 · see citing works
Strong and Simple Baselines for Multimodal Utterance Embeddings
Liang, P. P., Lim, Y. C., Tsai, Y.-H. H., Salakhutdinov, R., & Morency, L. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1267 · Cited by 29 · see citing works
Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling FusionPreprint
Mai, S., Xing, S., He, J., Zeng, Y., & Hu, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13572 · Cited by 26 · see citing works
Self-citationInvestigating Audio, Visual, and Text Fusion Methods for End-to-End Automatic Personality PredictionPreprint
Kampman, O. P., Barezi, E. J., Bertero, D., & Fung, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.00705 · Cited by 24 · see citing works
Frontal Low-rank Random Tensors for Fine-grained Action SegmentationPreprint
Zhang, Y., Muandet, K., Ma, Q., Heiko Neumann, B., & Tang, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01004 · Cited by 2 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Cited by
Sorted by citations
Mai, S., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3171679 · Cited by 182 · see citing works
Self-citationDisentanglement Translation Network for multimodal sentiment analysis
Zeng, Y., Yan, W., Mai, S., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102031 · Cited by 84 · see citing works
Self-citationLearning to Balance the Learning Rates Between Various Modalities via Adaptive Tracking Factor
Ya, S., Mai, S., & Hu, H. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3101421 · Cited by 65 · see citing works
Self-citationExcavating multimodal correlation for representation learning
Mai, S., Ya, S., Zeng, Y., & Hu, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.003 · Cited by 57 · see citing works
Self-citationLin, R., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3303711 · Cited by 56 · see citing works
Self-citationHuan, R., Zhong, G., Chen, P., & Liang, R. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3338769 · Cited by 53 · see citing works
Deng, H., Yang, Z., Hao, T., Li, Q., & Liu, W. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3211197 · Cited by 31 · see citing works
Multimodal Boosting: Addressing Noisy Modalities and Identifying Modality Contribution
Mai, S., Ya, S., Xiong, A., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3306489 · Cited by 29 · see citing works
Self-citationZeng, Y., Mai, S., & Hu, H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.109 · Cited by 31 · see citing works
Self-citationA text guided multi-task learning network for multimodal sentiment analysis
Luo, Y., Wu, R., Liu, J., & Tang, X. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126836 · Cited by 25 · see citing works
Chen, G., Qian, Z., Zhang, D., Qiu, S., & Zhou, R. (2025) · IEEE Access · doi:10.1109/access.2025.3544086 · Cited by 15 · see citing works
TriagedMSA: Triaging Sentimental Disagreement in Multimodal Sentiment Analysis
Luo, Y., Liu, W., Sun, Q., Li, S., Li, J., Wu, R., & Tang, X. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3524789 · Cited by 13 · see citing works
Yong, J., Wei, J., Lei, X., Wang, Y., Dang, J., & Lu, W. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102910 · Cited by 12 · see citing works
Yu, B., & Shi, Z. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06422-0 · Cited by 10 · see citing works
CAETFN: Context Adaptively Enhanced Text-Guided Fusion Network for Multimodal Sentiment Analysis
Li, J., Liu, R., Miao, Q., Wang, D., & Liu, X. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3590246 · Cited by 8 · see citing works
MulDeF: A Model-Agnostic Debiasing Framework for Robust Multimodal Sentiment Analysis
Huan, R., Zhong, G., Chen, P., & Liang, R. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521836 · Cited by 8 · see citing works
SGIR: Star Graph-Based Interaction for Efficient and Robust Multimodal Representation
Ding, Z., Lan, G., Song, Y., & Yang, Z. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3321404 · Cited by 5 · see citing works
FEF-Net: feature enhanced fusion network with crossmodal attention for multimodal humor prediction
Gao, P., Tao, C., & Guan, D. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01402-z · Cited by 4 · see citing works
Chen, G., Qian, Z., Qiu, S., Zhang, D., & Zhou, R. (2025) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105322 · Cited by 5 · see citing works
Generative AI for multimodal content: a survey with empirical and experimental evaluations
Taha, K. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11525-6 · Cited by 4 · see citing works
Multi-task disagreement-reducing multimodal sentiment fusion network
Zijun, W., Naicheng, J., Xinyue, C., & Sun, B. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105158 · Cited by 3 · see citing works
Xie, W., Liang, H., Niu, Z., Hou, X., Song, S., Yu, Z., & Shen, L. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3767746 · Cited by 3 · see citing works
Multi-Modal Graph-Based Sentiment Analysis via Hybrid Contrastive Learning
Zhang, J. (2024) · IEEE International Conference on Sensors, Electronics and Computer Engineering (ICSECE) · doi:10.1109/icsece61636.2024.10729439 · Cited by 3 · see citing works
Semantic-specific multimodal relation learning for sentiment analysis
Wu, R., Luo, Y., Liu, J., & Tang, X. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09644-8 · Cited by 3 · see citing works
Shen, Q., Xin, J., Dai, B. T., Huang, Z., & Wang, Z. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3639945
Zhou, H., Liu, J., Li, X., Liu, Y., & He, H. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3658336
- Authors
- Mai, S., Hu, H., & Xing, S.
- Affiliations
- Sun Yat-sen University (CN)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2022
- Volume
- 24
- Pages
- 2488-2501
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W3164173240
- Cited by
- 26
- References
- 74
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 7 · 2023: 6 · 2022: 3 · 2021: 2
- FWCI
- 1.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- Funders
- National Natural Science Foundation of China (62076262, 61673402, 61273270, 60802069)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Keywords
- multimodal fusion, facial expressions, nonverbal behavior, channel expansion, channel fusion, gating mechanism, social media videos
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
