The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Open accessAbstract
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On CMU-MOSEI dataset, our proposed C-MIB yields 0.7% improvement on Acc2 and 1.2% improvement on F1 score compared to the current state-of-the-art MISA [53]. The performance on Acc7 and Corr is also improved compared to the state-of-the-art methods.”
From Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations (2023) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 8×
“Inter-utterance context. et al. proposed one of the first models, bc-LSTM , which utilized”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“However, following recent works [7], including the stateof-the-art ICCN [50], we utilize the pre-trained BERT [11] as the feature extractor for textual utterances.”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“ndard low-level features that are provided by the respective benchmarks and utilized by the state-of-the-art methods. 4.3.1 Language Features. Traditionally, language modality features has been GloVe [38] embeddings for each token in the utterance. However, following recent works [7], including the stateof-the-art ICC…”
MethodInfluential(2008) · Journal of Machine Learning Research · Cited by 33,051
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 5×
“licly available models of [26, 51, 58]).”
BackgroundInfluentialSelf-citationEkman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,971 · cited 1×
“Both MOSI and MOSEI use Facet 3 to extract facial expression features, which include facial action units and face pose based on the Facial Action Coding System (FACS) [14].”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · cited 1×
“For UR_FUNNY, OpenFace [4], a facial behavioral analysis tool, is used to extract features related to the facial expressions of the speaker.”
Method
Research works printed but not linked: 15
- Andrew Galen, 2013, Proceedings of the 30th International Conference on Machine Learning, ICML 2013 (JMLR Workshop and Conference Proceedings, 1255
- Bousmalis Konstantinos, Advances in Neural Information Processing Systems 29. Curran Associates
- 10.5555/2354409.2355082 · DOI
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 19
- Drugman Thomas, 2011, Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonics. In INTERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association. ISCA
- Kiela Douwe, 2019, NeurIPS 2019 Workshop. Curran Associates, Inc.
- Lu Jiasen, Advances in Neural Information Processing Systems 32. Curran Associates
- 10.5555/2392963.2392965 · DOI
- Peng Minlong, 2018, Long Papers
- Rajagopalan Shyam Sundar, 2016, Proceedings, Part VII (Lecture Notes in Computer Science, 353
- Salzmann Mathieu, 2010, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010 (JMLR Proceedings, 708
- Hubert Tsai Yao-Hung, 2019, Learning Factorized Multimodal Representations. In 7th International Conference on Learning Representations, ICLR
- Vaswani Ashish, Advances in Neural Information Processing Systems 30. Curran Associates
- Zadeh Amir, 2018, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, 2246
- Zellinger Werner, 2017, Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning. In 5th International Conference on Learning Representations, ICLR
Cited by
Sorted by citations
Items 701–800 of 1,001
Hua, Q., Zhuo, C. H. E. N., Zhang, F., & Dong, C.-R. (2024) · JOURNAL OF SHENZHEN UNIVERSITY SCIENCE AND ENGINEERING · doi:10.3724/sp.j.1249.2024.03283
Active Exploration of Modality Complementarity for MultimodalSentiment Analysis
Xu, J., & Song, P. (2024) · Asia Conference on Computer Vision, Image Processing and Pattern Recognition (CVIPPR) · doi:10.1145/3663976.3663986 · Cited by 1
“Multimodal sentiment analysis (MSA) has attracted considerable research due to unprecedented advances in social media [9, 20, 24, 27].”
BackgroundMethodInfluentialResearch on Contrast-Based Negative Information Filtering in Multimodal Emotion AnalysisPreprint
Wang, R., Wang, Y., Feng, H., Yu, X., Guo, C., Zhao, Z., Huang, J., & Zhu, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4884270
sEMG-Based Multi-view Feature-Constrained Representation Learning
Yan, S., Dai, H., Wang, R., Zhang, L., & Wang, G. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5492-2_25
IG-GRD: A Model Based on Disentangled Graph Representation Learning for Imaging Genetic Data Fusion
Feng, S., Wang, L., Li, C., Wan, X., Zhang, F., & Hu, B. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5581-3_12
Enhancing Multimodal Emotion Analysis through Fusion with EMT Model Based on BBFN
Wang, W. (2024) · Transactions on Computer Science and Intelligent Systems Research · doi:10.62051/gfeqm854
“…Interaction Canonical Correlation Network (ICCN) [12], Multimodal Transformer (MulT) [13], Modality-Invariant and -Specific Representations (MISA) [14], Self-Supervised Multi-task Multimodal (Self-MM) [15], MultiModal InfoMax (MMIM) [16], Adaptive Multimodal Meta-Learning (AMML) [17],…”
MethodExtracting method for fine-grained emotional features in videos
Zheng, C., Peng, J., & Cai, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112382
Wave2Graph: Integrating spectral features and correlations for graph-based learning in sound waves
Hoang, T., Tran, T. K., Vu, X.-S., Nguyen, D.-K., Bhuyan, M. H., & Nguyen, H. D. (2024) · AI Open · doi:10.1016/j.aiopen.2024.08.004
Orthogonal Synthesis: Data Augmentation for Multimodal Sentiment Analysis
Zheng, T., & Liu, W. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650640
“Therefore, fine-tuning the pre-trained BERT over the sentiment analysis data achieves better performance than the non-BERT models [6]–[11].”
BackgroundMethodHajavi, A., Singh, H., & Fashandi, H. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650920
DRLN: Disentangled Representation Learning Network for Multimodal Sentiment Analysis
Hou, J., Omar, N., Tiun, S. B., Saad, S., & He, Q. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-7007-6_11
The Role of Multi-Modal Sentiment Analysis in Optimizing Leadership Communication
Khosla, A., & Gupta, G. (2024) · Advances in human and social aspects of technology book series · doi:10.4018/979-8-3693-4147-6.ch017
ConD2: Contrastive Decomposition Distilling for Multimodal Sentiment Analysis
Xi, Y., Huang, W., & Long, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8620-6_11
Zhao, X., Qu, L., Feng, T., Cai, J., & Tang, B. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681056 · Cited by 1
Modelling Social Intentions in Complex Conversational Settings
Kondyurin, I. (2024) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3678957.3688614
WConF: Weighted Contrastive Fusion for Multimodal Sentiment Analysis
Zeng, B., Li, R., Lu, L., Xie, L., Wang, J., Chen, W., & Deng, H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9443-0_3
Textual-dominated Multimodal Depression Detection
Huang, K. Z., Lu, H., & Li, J. (2024) · International Conference on Mechatronics and Machine Vision in Practice (M2VIP) · doi:10.1109/m2vip62491.2024.10746029
FINE-LMT: Fine-Grained Feature Learning for Multi-modal Machine Translation
Wang, Y., Zhang, Y., Li, D., Shen, J., Xu, Y., Xu, M., Funakoshi, K., & Okumura, M. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0119-6_32
Multimodal Sentiment Analysis Model Based on Text-Guided and Cross-Modal Feature Fusion
Wang, J., Zhao, T., Wang, Z., & Huang, Z. (2024) · International Conference on Robotics and Computer Vision (ICRCV) · doi:10.1109/icrcv62709.2024.10758615
“1) Unimodal Feature Encoder: Following previous work [5], [6], we use the pre-trained language model Bert to encode raw text tags as contextual word embeddings.”
Inter-Modal Shifting and Intra Adaptation for Multimodal Sentiment Analysis
Liu, K., Han, D., Zhao, D., Li, J., Qiao, B., & Wu, G. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0847-8_1
Adaptive Both homo- and hetero-Feature Integration for Multimodal Emotion Recognition
Wang, Z. K., & Jun, Z. (2024) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3696409.3700177
“To address these shortcomings, some studies explore the characteristics and commonalities of multimodal representations through feature disentanglement to promote more effective multimodal representation fusion [10] [17] [35] .”
Liu, Z., & Zhang, Z. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-9671-7_12
Wang, G., Huang, Y., & Zang, T. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821958
Embedding Speech Temporal Information in Pretrained Language Models for Sentiment Analysis
Liu, X., Wang, M., Huang, M., & Dong, C. (2024) · International Conference on Computer Science and Management Technology (ICCSMT) · doi:10.1145/3708036.3708088
Zhang, X., Zhao, Y., Liu, Y., & Zhang, D. (2024) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai62512.2024.00053
M3-Mixer:A Multimodal Mixer for Multimodal Sentiment Analysis with Multi-Head Loss
Hu, K., & Sun, D. (2024) · International Conference on Electronic Information Engineering and Computer Technology (EIECT) · doi:10.1109/eiect64462.2024.10866607
“MISA[14] learns both invariant and modality-specific features.”
Chakraborty, K., Vishwakarma, A. K., Krishna, M. M., Kumar, S., Kumar, P. A., & Anand, K. (2024) · International Conference on Multimedia Processing, Communication & Information Technology (MPCIT) · doi:10.1109/mpcit62449.2024.10892662
“In order to learn the similarities and features that are shared by unimodal representations, the modality-invariant and - specific representations (MISA) [10] incorporates a number of different losses.”
The Potential of Speech Features to Discriminate between Original and Machine-Translated Texts
Chen, Y., Farrús, M., & Toral, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887578
“…architecture of multimodal stress detection using emotion recognition (MUSER) [12], tensor fusion (TF) [25], and modality-invariant and specific representations for multimodal sentiment analysis (MISA) [26], as follows: • MUSER : Unimodal features are separately learned and subsequently integrated.”
Research on crime motivation identification and quantitative analysis methods based on EEG signals
Ma, D. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1544589
RetractedFactorization-Based Information Reconstruction for Enhancing Missing Modality Robustness
Wang, C., Zhou, M., Yu, X., Weng, J., & Zhang, Y. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3552435
“While substantial progress has been made in MSA under the assumption that all modalities are fully available during both training and inference [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], this ideal scenario rarely holds true in real-world applications.”
Multimodal sentiment analysis and fusion based on deep learning
Li, Y., Xu, Y., & Li, Y. (2025) · International Conference on Big Data, Computational Intelligence, and Applications (BDCIA) · doi:10.1117/12.3059330
“MISA [13]: A structure for extracting mode-agnostic features and mode-specific representations in multimodal sentiment analysis is introduced.”
Self-Supervised Text-Centric Sentiment Analysis via Sparse Cross-Modal Interactions
Zhou, H., Liu, J., Li, X., & Jiao, J. (2024) · International Conference on Computer, Big Data and Artificial Intelligence (ICCBD+AI) · doi:10.1109/iccbd-ai65562.2024.00021
A Transfer Learning-Based Text-Centric Model for Multimodal Sentiment Analysis
Shaowei, Y. I., & Zainudin, S. (2025) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.1548
RoLiVit: Feature Fusion Approach for Multimodal Sentiment Analysis Using Deep Learning
Shroff, N., Patel, S., & Shah, H. (2025) · SN Computer Science · doi:10.1007/s42979-025-03906-5
Zhang, Z., Yang, R., & Zhang, C. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-5815-2_12
Decoupled Representation with Multimodal Prompts for Emotion Recognition in Conversation
Dong, Q., Xiang, J., Yu, G., Ren, W., Wang, Z., Wang, J., & Liu, H. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-5084-2_7
Multimodal Sentiment Analysis Using TriFusion Attention Network
Prasika, L., Dinesh Kumar, S., & Manoj, S. (2025) · International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET) · doi:10.1109/wispnet64060.2025.11005330
Multimodal Sentiment Analysis Based on Hierarchical Distance-Aware Contrastive Learning
吕, 欣. (2025) · Computer Science and Application · doi:10.12677/csa.2025.155134
Sentiment analysis for live video comments with variational residual representations
Luo, C., Fang, L., & Qiu, B. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101838
An Enhanced Feature Fusion Network for Multimodal Sentiment Analysis
Xu, Y. L., & Wu, Y. (2024) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait63902.2024.11022222
Multimodal Sentiment Analysis Model Based on Multi-Task Stacking TCN and Transformer
Qiu, X., Xin, Z., Luo, Y., & Sun, Z. (2024) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait63902.2024.11022242
“MISA [17] adopts a subspace learning approach to map modalities into different subspaces, enhancing multimodal representation.”
Liu, J., & Yang, L. (2024) · International Conference on Virtual Reality and Visualization · doi:10.1109/icvrv62410.2024.00039
Text Enhancement-Based Multimodal Fusion for Video Sentiment Analysis
Luo, J., Zhu, Y., & Peng, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8298-0_13
QSMT: Query-Shared Multimodal Transformer for Multimodal Sentiment Analysis
Liu, H., Liu, Y., Ding, J., & Zhang, W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8180-8_9
Deep Fusion for Deeper Feelings: Uniting Visual and Textual Cues in Emotion Detection
Vijayenth, R. V., Sundarambal, B., Kirubakaran, N., Senthil Kumar, G., Selvaganesan, C., & Senthil Kumar, S. (2025) · International Conference on Data Science and Business Systems (ICDSBS) · doi:10.1109/icdsbs63635.2025.11031856
Hariharan, R., & Gideon, D. A. M. (2025) · International Conference on Augmented Intelligence and Sustainable Systems (ICAISS) · doi:10.1109/icaiss61471.2025.11041762
“Spectrogram is used to transmit the audio signal into spectral information with amplitude and sampling rate[10].”
LGCMNet: Multimodal Sentiment Analysis Network Based on Language-Guided Cross-Modal Interaction
Wang, Y., Nuo, M., Zhang, Y., & Jia, X. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6599-0_11
DCLMA: Deep correlation learning with multi-modal attention for visual-audio retrieval
Zhang, J., & Hachiya, H. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100695
Mitigating modality bias in multi-label emotion recognition
Ge, S., & Chen, Y. H. (2025) · IISE Transactions · doi:10.1080/24725854.2025.2531044
“MISA (Multi-modal Interconnected Semantic Attention) (Hazarika et al., 2020): this model projects each modality into common modality and specific modality representation spaces.”
Motif: A Text-Aware and Disentangled Mamba-Based Architecture for Multimodal Emotion RecognitionPreprint
Hu, B., & Li, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5351685
Zi, L., Lv, H., & Cong, X. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9946-9_31
When Sentiment Analysis Faces Missing Modalities: A Specific and Invariant Feature Learning Approach
Zhan, Z., Cao, D., & Cheng, H. (2025) · International Conference on Neural Networks, Information and Communication Engineering (NNICE) · doi:10.1109/nnice64954.2025.11064735
Auxiliary Context Module and Weighted Multihead Fusion for Multimodal Intent Recognition
Xia, Y., Song, J., & Tian, S. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9994-0_9
Multimodal Polarity-Semantic Coupling Network for Sarcasm Analysis
Tang, J., Yang, Z., Zhou, F., Ozawa, K., Toya, T., & Kong, W. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-7008-6_9
Qian, B., Ren, Q.-D.-E.-J., & Ji, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0020-8_11
Qian, B., Ren, Q.-D.-E.-J., Peng, W., & Ji, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0020-8_1
DSMRec: Dual-Spectrum Mamba Based Model for Sequential Recommendation
He, R., Liu, K., Shang, P., Hong, Y., & Cai, D. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9881-3_41
Sentiment analysis based on self-supervised learning cross-modal data augmentation
Cao, R., Zhang, Y., & Chen, Z. (2025) · International Conference on Electronic Engineering and Information Systems (EEISS) · doi:10.1109/eeiss65394.2025.11085602
“The MISA model proposed by Devamanyu et al. introduces an innovative framework for multimodal sentiment analysis, where the features of each modality are mapped to two separate feature spaces, improving the capability of feature representation [4].”
Sentiment Perception from Tokens: A Multitask Learning Framework with Entropy-Driven Fusion
Yan, J., Li, A., Xu, L., & Wang, P. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0009-3_23
Xu, F., Bi, H., Li, B., Du, Q., & Cai, Q. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0014-7_11
MICTE: Mutual Information and Cross-Modal Text Enhancement for Multimodal Sentiment Analysis
Ma, J., Li, X., Ding, N., Huan, R., & Zhao, X. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9911-7_4
Zhu, L., Peng, J., & Zhang, H. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9986-5_14
Audio Sentiment Analyzer for Mental Health Support Hotline
Maurya, S., Saxena, S., Shukla, S., Maurya, S., & Verma, A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-3942-7_45
Zhang, K., Pei, H., Zhang, C., & Pang, S. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3593333 · Cited by 1
Sun, L., Zhu, K. X., & Zhao, C. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9921-6_6
CAtCh: Cognitive Assessment through Cookie Thief
Colonel, J. T., Hagler, C., Wismer, G. A., Curtis, L. M., Becker, J., Wisnivesky, J. P., Federman, A. D., & Pandey, G. S. (2025) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh67620.2025.00029
Mallia, B. (2025) · Artificial intelligence · doi:10.5772/intechopen.1010889
Adaptive Multi-Dimensional and Multi-Scale Convolutional Fusion for Multimodal Sentiment Analysis*
Xiao, H., Liu, S., Wang, H., Cai, H., & Ni, S. (2025) · IEEE International Conference on Real-time Computing and Robotics (RCAR) · doi:10.1109/rcar65431.2025.11139527
Multimodal Sentiment Analysis with Parallel Attention and Correlation Fusion
Liu, X., Lei, J., Wu, J., Feng, Z., & Liang, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04549-2_6
Hierarchical Multi-Modal Transformer for Cross-Modal Long Document Classification
Liu, T., Hu, Y., Gao, J., Sun, Y., & Yin, B. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3608295 · Cited by 3
“(cid:2) MISA [49]: Combining modality-invariant and modality-specific features to predict emotional states.”
MCFM: Multimodal Competitive Fusion Mechanism for Sentiment Analysis
Xing, M., Zhai, Z., Deng, M., Cai, Q., Ren, H., & Wang, T. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3598637
“Hazarika et al. [28] proposed MISA (modality-invariant and -specific representations), which projected each modality into two different subspaces.”
Beyond Utterance: Understanding Group Problem Solving through Discussion Sequences
Duan, Z., Yang, Z., Welles, B. F., & Radke, R. J. (2025) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3716553.3750758
“Emotion recognition is addressed in both human-human and human-agent settings [16, 19], while intent modeling is enhanced through contrastive multimodal techniques [40].”
DiffuFuse: Diffusion-Driven Dual-Stream Fusion Framework for Multimodal Sentiment Analysis
Lv, X., Wen, Y., & Yu, H. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755508 · Cited by 3
A Theoretical Proof of Dynamic Multimodal Fusion Exacerbates Modality Greedy
Ding, X., Ma, H., & Zhang, C. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3754820
ATD-AMSMamba: Improving Robustness of State Space Models for Multimodal Sentiment Analysis
Li, Y., Dong, Z., Fang, Z., & Li, L. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209183
Decoding Emotional Silences: Reliable Multimodal Sentiment Analysis with Bipolar Uncertainty
Wei, Y., Fu, H., Li, Y., Xin, Y., Xu, X., Zhou, F., & Zhong, T. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209057
“In this study, we compare our method with eight baseline models as below: MISA [18], MMIN [7], Self-MM [19], TFR-Net [10], CENET [20], TETFN [21], ALMT [13] and LNLN [9].”
Fan, J., Chen, W., Fan, Z., & Yu, R. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209913
MAMF-Net: Modality-Adaptive Masked Fusion Network for Speech Emotion Recognition
Li, H., Lu, T., Wang, J., Chen, X., Zhang, Y., & Liu, S. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209516
“Modality independence and feature fusion models, featuring EMT [26], which strengthens modality independence via transformer modules, and MISA [27], which optimizes data fusion through modality-specific and invariant subspaces.”
A Multi-Grained Perception Model for Sentiment Analysis with Perceived Contrastive Focal Loss
Jin, W., Lin, J., Yang, Z., Xie, H., Yu, F., & Li, X. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11210018
Towards Trustworthy Model via Uncertainty Verification in Multimodal Sentiment Analysis
Tang, C., Li, Y., Shen, T., Gong, X., & Zhang, T. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11208957
Niu, T., Tu, G., Wang, H., Qin, B., & Xu, R. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11208939 · Cited by 1
Prompt-Guided Multi-Task Decoupling for Speech Presentation Skills Assessment
Xiong, Z., Tan, J., Zhang, T., Wu, B., & Zheng, C. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209836
EPIC: Efficient Prompt Interaction for Text-Image Classification
Yu, X., Sun, H., Ling, Z., Niu, Z., Bai, Z., Qin, R., Chen, Y.-W., & Lin, L. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11210028
Wang, Y., Wang, K., Zhang, H., Fang, M., & Liu, S. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11210194
“TriMod-EFN is compared with several baseline models to verify its effectiveness, including unimodal methods (e.g., BERT T for text, Wav2vec A for audio, R-CNN V for visual), as well as multimodal methods (e.g., Trans TAV , MAG-BERT [22], MulT [23], MISA [24], EMRFM [4], and TCL-MAP [25]).”
Feature Decoupling with Modality Modulation for Multimodal Sentiment Analysis
Wang, Y., Wang, J., Zheng, A., Cheng, W., & Sheng, X. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3393-0_8
Leelavathy, B., Chenoori, R. K., Krishna, P. V., Kovvur, R. M. R., Harshavardhan, B., & Mirza Rafiq, A. (2025) · IEEE World Conference on Applied Intelligence and Computing (AIC) · doi:10.1109/aic66080.2025.11212000
“Adaptive Language-guided Multimodal Transformer (ALMT) is a complex model that includes Adaptive Hyper-modality learning Layer (AHL) [8] for learning and analysing from the embeddings whose base is obtained from the MISA [9].”
Zhao, X., Yang, H., Wang, D., Tang, B., & Zheng, Y. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3630478
“…with Tensor Fusion Network (TFN) [59], the bidirectional LSTM/1D CNN-based model (BiLSTM-1DCNN) [60], Multimodal Transformer (MulT) [61], MISA [62], D-vlog [27], bidirectional contextual LSTM (bc-LSTM) [63], Emotion Recognition [64], Sequence Modeling [65], Topic Modeling [66], Context-aware…”
Zhao, Y., & Zhang, M. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4088-4_13
DGFN: Brain-Inspired Distributed Fusion for Robust Multimodal Sentiment Analysis
Li, N., Zhang, R., Zhou, X., Dai, W., Qi, J., & Zhou, Y. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4091-4_2
Wang, C., Liang, J., Cao, P., Yang, J., & Zai͏̈ane, O. R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5745629
Wang, M., & Jiang, H. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5743545
MAGI: Modality-Aligned Geometry-Aware Integration for Robust Multimodal Sentiment Analysis
Hu, B., & Li, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3349-7_46
DiGTF: A Difference-Guided Two-Stage Fusion Framework for Multimodal Sentiment Analysis
Liu, H., Nuo, M., Li, R., & Zhou, C. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3349-7_30
Multi-channel Cross-modal Fusion With Contrastive Learning For Multimodal Sentiment Analysis
Zhang, D., Zhang, S., Ran, N., & Wang, L. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228144
“MSA methods can be classified into two main categories: representation learning-centered approaches [1]–[4] and multimodal fusion-focused approaches [5]–[7].”
Modality Modulation with Adaptive Fusion for Multimodal Sentiment Analysis
Zheng, A., Wang, Y., Wang, J., Sheng, X., & Cheng, W. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228399
Xu, F., Bi, H., Li, B., Wu, Z., & Jiafu, M. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229219
“These baselines include LF-DNN [11], MFN [12], LMF [13], TFN [7], MulT [14], MISA [4], MAG-BERT [15], HyCon [9], Self-MM [16], MUTA [19], SKEAFN [18], PS-Mixer [17], CAGC [20], CLGSI [22], and VLP2MSA [21].”
HACL: A Hybrid Adaptive Curriculum Learning Framework for Multi-modal Sarcasm Detection
Shen, K., Huang, Y., Wang, W., Xie, S., Zhu, Z., & Liu, W. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228306
Modality Modulation with Adaptive Fusion for Multimodal Sentiment Analysis
Zheng, A., Wang, Y., Wang, J., Sheng, X., & Cheng, W. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229254
GExSent: Gated Experts for Robust Sentiment Analysis Across Modalities
Sethi, S., Saini, D., Singh, S., Dhiman, C., & Bhushan, B. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228437
Shen, L., Zhang, Q., Li, W., Lu, M., & Liang, X. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3052-6_5
Metadata
- Authors
- Hazarika, D., Zimmermann, R., & Poria, S.
- Affiliations
- National University of Singapore (SG)
- Singapore University of Technology and Design (SG)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 1122-1131
- OpenAlex
- W3093051361
- Cited by
- 1,073
- References
- 45
- Citations by year
- 2026: 280 · 2025: 315 · 2024: 238 · 2023: 136 · 2022: 78 · 2021: 23 · 2020: 3
- FWCI
- 22.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- Funders
- Ministry of Education - Singapore (MOE2018-T2-1-103)
- DSO National Laboratories - Singapore (RTDST190702)
- RIE 2020 Advanced Manufacturing and Engineering (AME) programmatic grant (A19E2b0098)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, modality gap, modality-invariant representations, modality-specific representations, subspace projection, multimodal fusion, user-generated video, humor detection, MOSI, MOSEI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
