The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Open accessAbstract
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On CMU-MOSEI dataset, our proposed C-MIB yields 0.7% improvement on Acc2 and 1.2% improvement on F1 score compared to the current state-of-the-art MISA [53]. The performance on Acc7 and Corr is also improved compared to the state-of-the-art methods.”
From Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations (2023) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 8×
“Inter-utterance context. et al. proposed one of the first models, bc-LSTM , which utilized”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“However, following recent works [7], including the stateof-the-art ICCN [50], we utilize the pre-trained BERT [11] as the feature extractor for textual utterances.”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“ndard low-level features that are provided by the respective benchmarks and utilized by the state-of-the-art methods. 4.3.1 Language Features. Traditionally, language modality features has been GloVe [38] embeddings for each token in the utterance. However, following recent works [7], including the stateof-the-art ICC…”
MethodInfluential(2008) · Journal of Machine Learning Research · Cited by 33,051
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 5×
“licly available models of [26, 51, 58]).”
BackgroundInfluentialSelf-citationEkman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,971 · cited 1×
“Both MOSI and MOSEI use Facet 3 to extract facial expression features, which include facial action units and face pose based on the Facial Action Coding System (FACS) [14].”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · cited 1×
“For UR_FUNNY, OpenFace [4], a facial behavioral analysis tool, is used to extract features related to the facial expressions of the speaker.”
Method
Research works printed but not linked: 15
- Andrew Galen, 2013, Proceedings of the 30th International Conference on Machine Learning, ICML 2013 (JMLR Workshop and Conference Proceedings, 1255
- Bousmalis Konstantinos, Advances in Neural Information Processing Systems 29. Curran Associates
- 10.5555/2354409.2355082 · DOI
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 19
- Drugman Thomas, 2011, Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonics. In INTERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association. ISCA
- Kiela Douwe, 2019, NeurIPS 2019 Workshop. Curran Associates, Inc.
- Lu Jiasen, Advances in Neural Information Processing Systems 32. Curran Associates
- 10.5555/2392963.2392965 · DOI
- Peng Minlong, 2018, Long Papers
- Rajagopalan Shyam Sundar, 2016, Proceedings, Part VII (Lecture Notes in Computer Science, 353
- Salzmann Mathieu, 2010, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010 (JMLR Proceedings, 708
- Hubert Tsai Yao-Hung, 2019, Learning Factorized Multimodal Representations. In 7th International Conference on Learning Representations, ICLR
- Vaswani Ashish, Advances in Neural Information Processing Systems 30. Curran Associates
- Zadeh Amir, 2018, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, 2246
- Zellinger Werner, 2017, Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning. In 5th International Conference on Learning Representations, ICLR
Cited by
Sorted by citations
Items 301–400 of 1,001
Towards Multimodal Prediction of Spontaneous Humor: A Novel Dataset and First Results
Christ, L., Amiriparian, S., Kathan, A., Müller, N., Konig, A. S., & Schuller, B. W. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3475736 · Cited by 32
Chen, Z. J., Zhu, Z., Xu, W., Zhang, Y., Wu, X., & Zheng, Y. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681189 · Cited by 9
Zheng, W., Yu, J., & Xia, R. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681638 · Cited by 12
Sun, Q., Zuo, H., Liu, R., & Li, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3490694 · Cited by 9
“Such operations ignore the complex inherent correlations and heterogeneity among different modalities [12], [13], thus limiting the MSA performance.”
Global distilling framework with cognitive gravitation for multimodal emotion recognition
Li, H., Zhong, H., Xu, C., Liu, X., Wen, G., & Liu, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2024.129306 · Cited by 9
Zhang, Y., Zhong, H., Alhusaini, N., Chen, G., & Wu, C. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113121 · Cited by 9
Qin, J., Liu, F., & Zong, L. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107449 · Cited by 9
Multi-modal sentiment recognition with residual gating network and emotion intensity attention
Wang, Y., Guo, X., Hou, X., Miao, Z., Yang, X., & Guo, J. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107483 · Cited by 9
SAGE-Net: Single-layer augmented gated encoder network for efficient multimodal sentiment analysis
Zhou, J., Kang, X., Ding, W., Wang, L., Ding, F., Matsumoto, K., Zhang, C., & Chi, H. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113665 · Cited by 10
Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment Analysis
Luo, M., Jiang, Y., & Mai, S. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755591 · Cited by 19
Liu, R., Zuo, H., Lian, Z., Yuan, H., & Fan, Q. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755605 · Cited by 9
Multimodal Classification and Out-of-Distribution Detection for Multimodal Intent Understanding
Zhang, H., Zhou, Q., Xu, H., Su, J., Evans, R. H., & Gao, K. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3618541 · Cited by 19
“MISA [8] separates each modality into modality-invariant and modality-specific subspaces for fine-grained fusion.”
Quan, Z., Sun, T., Su, M., Wei, J., Zhang, X., & Zhong, S. (2022) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53654.2022.9945514 · Cited by 10
“Hazarika et al. [5] designed a scheme to learn modality invariant and modality-specific representations, combining regularization to help learn both types of modality representations.”
BackgroundInfluentialMMATR: A Lightweight Approach for Multimodal Sentiment Analysis Based on Tensor Methods
Koromilas, P., Nicolaou, M. A., Γιαννακόπουλος, Θ., & Panagakis, Y. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097030 · Cited by 9
“…models the parameters of the unimodal feature extraction networks (e.g., BERT) are not included in the final number of parameters, since it is a common practice [12, 10, 13, 14, 15, 16, 17, 4, 8, 2, 3, 1] to use pre-extracted features in order to evaluate the performance on multimodal sentiment.”
BackgroundMethodInfluentialMulti-modal anchor adaptation learning for multi-modal summarization
Chen, Z., Lu, Z., Rong, H., Zhao, C., & Xu, F. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127144 · Cited by 9
FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wild
Liu, Z.-S., Courant, R., & Kalogeiton, V. S. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02000-2 · Cited by 13
VAD: A Video Affective Dataset With Danmu
Wang, S., Li, X., Zheng, F., Pan, J., Li, X., Chang, Y., Zhu, Z., Li, Q., Wang, J., & Xiao, Y. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3382503 · Cited by 9
MMTrans-MT: A Framework for Multimodal Emotion Recognition Using Multitask Learning
Shen, J., Zheng, J., & Wang, X. (2021) · International Conference on Advanced Computational Intelligence (ICACI) · doi:10.1109/icaci52617.2021.9435906 · Cited by 9
“Considering the high representation capabilities of words modality, some works utilized self-attention and migrated Transformer-based model into multimodal fusion [14–17].”
BackgroundMethodLanguage Reinforced Superposition Multimodal Fusion for Sentiment Analysis
He, J., & Hu, H. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3180668 · Cited by 10
TEASEL: A Transformer-Based Speech-Prefixed Language ModelPreprint
Arjmand, M., Dousti, M. J., & Moradi, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05522 · Cited by 21
“• Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (MISA) factor-izes modalities into modality-invariant and modality-specific features and then fuses them to predict a label [42].”
BackgroundMethodInfluentialScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally DescriptorsPreprint
Luo, H., Ji, L., Huang, Y., Wang, B., Ji, S., & Li, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.01368 · Cited by 20
“It proves the effectiveness of the S3C loss on representation Methods MV-LSTM (Rajagopalan et al., 2016) 73.9/- 74.0/- 1.019 0 .601 TFN (Zadeh et al., 2017) 73.9/- 73.4/- 1.040 0 .633 MARN (Zadeh et al., 2018c) 77.1/- 77.0/- 0.968 0.625 MFN (Zadeh et al., 2018a) 77.4/- 77.3/- 0.965 0.632 RMFN (Liang et al., 2018) 78.4…”
InfluentialPCENet: Psychological Clues Exploration Network for Multimodal Personality Assessment
Zhu, Y., Wei, Y., Li, M., Zhang, T., Wei, S., & Wu, B. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615005 · Cited by 8
Flexible Material Quality Assessment Based on Visual–Tactile Fusion
Xu, S., Xu, H., Mao, F., Su, W., Ji, M., Gan, H., & Yang, W. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3386205 · Cited by 8
Semantic-wise guidance for efficient multimodal emotion recognition with missing modalities
Liu, S., Wang, Y., Wang, K., Li, B., Yang, F., & Yang, S. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01310-2 · Cited by 8
“Hazarika et al. [12] proposed the Modality-Invariant and Modality-Specific Representations for Multimodal Sentiment Analysis (MISA) model, which projects each modality feature into a shared semantic space to identify potential commonalities among modalities.”
GRACE: GRadient-based Active Learning with Curriculum Enhancement for Multimodal Sentiment Analysis
Li, X., Ye, W., Zhang, Y., & Sun, X. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681617 · Cited by 11
Dynamic Weighted Gating for Enhanced Cross-Modal Interaction in Multimodal Sentiment Analysis
Wang, N., & Wang, Q. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3702996 · Cited by 9
“MISA [34]: Each modality is mapped to two different representation spaces, one space learns the commonalities between different modes and gradually ranges the gap between modes, the other space learns the unique characteristics of modes, and finally integrates the two representation spaces.”
Rahman, M., Provath, M. A.-M., Deb, K., Dhar, P. K., & Shimamura, T. (2025) · IEEE Access · doi:10.1109/access.2025.3535694 · Cited by 8
“Recent works have also been inclined towards the fusion of multi-modal information using multi-modal fusion techniques, such as MISA [9] and UR-FUNNY [8], which exhibit a significant gain and achieve more than 80% accuracy in humor sarcasm detection on more than one modality.”
InfluentialMamba-assisted modality subspace complementary fusion for multimodal sentiment analysis
Chen, Q., Tang, Y., & Liu, H. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.03.021 · Cited by 9
CAETFN: Context Adaptively Enhanced Text-Guided Fusion Network for Multimodal Sentiment Analysis
Li, J., Liu, R., Miao, Q., Wang, D., & Liu, X. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3590246 · Cited by 9
“The key challenge in MSA is how to design an efficient multimodal fusion method that comprehensively considers differences and interactions between heterogeneous modalities [5].”
InfluentialChen, J., Zhong, Y., Huang, Q., Huang, C., Jiang, F., Huang, X., & Wang, X. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3606964 · Cited by 9
“…[11], [12], [13], [14], [15], [16], [17], [18] or Transformer-based models [19], [20], [21], [22], [23] to integrate different modalities, while others design self-supervised learning methods [24], [25], [26] to reduce the gap between modalities to facilitate modality alignment and fusion.”
InfluentialLi, H., Han, H., Xu, C., Chen, T., Liu, X., & Wen, G. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114127 · Cited by 11
Transformer-Based Physiological Feature Learning for Multimodal Analysis of Self-Reported Sentiment
Katada, S., Okada, S., & Komatani, K. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556576 · Cited by 8
“Although several Transformer-based multimodal models for affective computing and sentiment analysis have recently been proposed [4, 8, 22], a Multimodal transformer called MulT was the frst model proposed in multimodal sentiment analysis research [27].”
BackgroundMethodHuang, Q., Cai, P., Nie, T., & Zeng, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446825 · Cited by 8
“…methods based on learning: TFN [4], LMF [17] and MFM [18]; Works based on fusion: MulT [5] and MAG-BERT [19]; The methods based on feature space: MISA [2]; Multi-task-based works: Self-MM [16] and TPMSA [20]; And the methods based on contrastive learning: MMIM [21], HyCon [3], and WSCL-CL [22].”
MethodResultLi, Y., Weng, W., & Liu, C. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09634-w · Cited by 8
MT-TCCT: Multi-task Learning for Multimodal Emotion Recognition
Wang, Y., Chen, Z., Chen, S., & Zhu, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15934-3_36 · Cited by 8
Towards Temporal Event Detection: A Dataset, Benchmarks and Challenges
Yang, Z., Yang, Z., Guo, Z., Lin, Z., Zhu, H., Li, Q., & Liu, W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3276523 · Cited by 9
“More specifically, joint-embedding-based fusion methods include MCTN [45], MFM [66], MISA [46] and MIB [47], sequential network-based fusion meth-ods include EF_LSTM [67], LF_DNN [48], Graph-MFN [68] and MFN [49], tensor-based fusion methods include LMF [51] and TFN [50], transformer-based fusion…”
BackgroundMethodA Comprehensive Survey on Multi-Modal Conversational Emotion Recognition with Deep LearningPreprint
Shou, Y., Meng, T., Ai, W., Yin, N., & Li, K. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5017731 · Cited by 8
Hierarchical Knowledge Stripping for Multimodal Sentiment Analysis
Xiong, A., Zeng, Y., & Hu, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3456117 · Cited by 8
“Recently, feature space manipulation-based fusion has attracted more attention, and researchers focus on performing a series of mathematical operations on the feature space to learn the interaction relationship between various modalities [14], [24], [26], [27].”
Chen, Q., Xie, S., Fang, X., & Sun, Q. (2024) · The Visual Computer · doi:10.1007/s00371-024-03668-w · Cited by 7
Mutual Information of Crossmodal Utterance Representation for Multimodal Sentiment Analysis
Xufei, Yue, D., & Wei, X. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3466968 · Cited by 9
“Inspired by MISA [3] and Mutual Information (MI) [4], [5], we propose Mutual Infomax Utterance Representation (MIUR), a novel structure for information exchange between each utter-ance representation before multimodal fusion.”
InfluentialMulDeF: A Model-Agnostic Debiasing Framework for Robust Multimodal Sentiment Analysis
Huan, R., Zhong, G., Chen, P., & Liang, R. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521836 · Cited by 8
“Here, we adopt three base MSA models: MAG-BERT [4], MISA [5], and Self-MM [38].”
InfluentialPrivacy-Preserving Multimodal Sentiment Analysis
Xu, H., Li, W., Takabi, D., Seo, D., & Cai, Z. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3527864 · Cited by 7
“The decoder D is defined in (7) to ensure that the encoded representations indeed represent the details of the corresponding modality data [4], [49] where f m is the reconstructed feature vector for m ∈ { v , a , l } , and θ d represents the parameters of the decoder D .”
InfluentialCompleted Feature Disentanglement Learning for Multimodal MRIs Analysis
Liu, T., Liu, H., Shang, F., Yu, L., Han, T., & Wan, L. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3539712 · Cited by 9
“3) Compared Methods: We compare the proposed framework with nine state-of-the-art (SOTA) MML methods, including EmbraceNet [21], ETMC [16], ADCCA [22], MAML [18], NestedFormer [19], MISA [8], DMD [10], CCML [17] and GLoMo [23].”
InfluentialMulti-level Conflict-Aware Network for Multi-modal Sentiment Analysis
Gao, Y., Wu, H., & Zhang, L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888597 · Cited by 7
“In recent years, multimodal sentiment analysis (MSA) has attracted increasingly widespread attention [1, 2, 3, 4].”
Li, Z., Huang, Z., He, X., Yu, J., Chen, H., Yang, C., & Pan, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113376 · Cited by 7
Wang, M., Xie, L., Li, C., Wang, X., Sun, M., & Liu, Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127631 · Cited by 8
HGTFM: Hierarchical Gating-Driven Transformer Fusion Model for Robust Multimodal Sentiment Analysis
Yang, C., Liang, Z., Yan, D., Hu, Z., & Wu, T. (2025) · IEEE Access · doi:10.1109/access.2025.3560641 · Cited by 7
“This method is the basis of the improved model in this chapter, through The Transformer model constructs the modal conversion task to discover cross-modal interaction relationships and learn fusion feature representations MISA [13].”
Chen, M., Liu, J., Li, X., Zhang, Y., Liu, H., Jiao, J., & He, H. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3569089 · Cited by 7
“Representation learning-based methods, such as MISA [17] and Self-MM, aim to model both the consistent and divergent information across modalities to improve MSA performance.”
CIEG-Net: Context Information Enhanced Gated Network for multimodal sentiment analysis
Chen, Z., Lu, C., & Wang, Y. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111785 · Cited by 8
Sun, N., Zhang, W., Zheng, W., Liu, J., Chai, L., & Wu, C. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3612369 · Cited by 7
“In addition to the eight baseline methods [9], [17], [18], [19], [20], [21], [24], [25] introduced in [35], we also include DMD , EMT-DLFR , NIAT , CRNet , Meta-NA , TSCL-FHFN , and TSST .”
He, X., Pan, Y., Xu, Z., Li, Z., Guo, X., & Yang, C. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3614159 · Cited by 7
“The disentanglement of modal information has emerged as a prominent approach, with MISA [18] proposing a shared encoder in conjunction with three dedicated encoders to map modalities into modal-invariant and modal-specific subspaces.”
NeuProNet: neural profiling networks for sound classification
Tran, K.-T., Vu, X.-S., Nguyen, K., & Nguyen, H. D. (2024) · Neural Computing and Applications · doi:10.1007/s00521-023-09361-8 · Cited by 7
Circular Decomposition and Cross-Modal Recombination for Multimodal Sentiment Analysis
Liang, H., Xie, W., He, X., Song, S., & Shen, L. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446166 · Cited by 7
“To bridge modality gaps, certain MSA models [9, 10, 11] decouple each modal representations to modality-invariant and modality-specific features to extract the consistency and distinctiveness of heterogeneous modalities.”
BackgroundMethodInfluentialMMLSCU: A Dataset for Multi-modal Multi-domain Live Streaming Comment Understanding
Meng, Z., Gao, Q., Guo, D., Li, Y., Li, B., Fei, H., Wu, S., Li, F., Teng, C., & Ji, D. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645677 · Cited by 9
Yang, S., Cai, J., & Liao, L. (2024) · Journal of Intelligent Information Systems · doi:10.1007/s10844-024-00858-9 · Cited by 7
“MISA (Hazarika et al., 2020) Each modality in MISA is projected into two distinct subspaces, decomposing the modality into modality-invariant features and modality-specific features, which are then fused to predict the sentiment state.”
Mu, G., Chen, C., Li, X., Li, J., Ju, X., & Dai, J. (2024) · IEEE Access · doi:10.1109/access.2024.3493933 · Cited by 7
FunnyNet: Audiovisual Learning of Funny Moments in Videos
Liu, Z.-S., Courant, R., & Kalogeiton, V. S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26316-3_26 · Cited by 7
“4 - - - - - MISA 2020 (V+A+T) [14] - - - - - 66.”
MethodInfluentialLearning modality-fused representation based on transformer for emotion analysis
Shi, P., Hu, M. Q., Ren, F., Shi, X., & Xu, L. (2022) · Journal of Electronic Imaging · doi:10.1117/1.jei.31.6.063032 · Cited by 7
“For the CMU-MOSEI dataset, except for Acc2 being slightly lower than MISA, the other indicators are all superior to the baseline models mentioned in this paper.”
BackgroundMethodDas, R., & Singh, T. D. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15356-3 · Cited by 7
“Hazarika et al. [15] proposed a novel multimodal framework named MISA that learns factorized subspaces for each modality and provides better representations as input to fusion.”
MECG: modality-enhanced convolutional graph for unbalanced multimodal representations
Tang, J., Ni, B., Yang, Y., Ding, Y., & Kong, W. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06729-y · Cited by 6
“� � That is, M a = Conv1D( M a ), and Compared to previous work [9, 25], we use two bidirectional LSTM [29] to capture the time characteristics of these modes instead of one-way LSTM. Specifically, we use bidirectional long short-term The acoustic and visual feature calculation with the guidance of…”
InfluentialHu, Z., Yu, Z., & Guo, J. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01697-6 · Cited by 6
“MISA [17] maps each modal feature to two different representation spaces and learns both modality-invariant and modality-specific representations.”
Federated Deep Reinforcement Learning for Multimodal Content Caching in Edge-Cloud Networks
Feng, W., Zuo, X., Zhang, R., Zhu, Y., Wang, C., Guo, J., & Sun, C. (2025) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2025.3545924 · Cited by 6
“Before fusing the various aspects, it is important to address the fundamental gaps in knowledge that exist across distinct modalities [34].”
Wang, X., Wang, L., Yu, Y., & Jiao, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888606 · Cited by 8
“For example, in [19]– [21], researchers mitigated modal heterogeneity by decoupling multimodal features into modality-invariant and modality-specific features and using loss constraints.”
A method for multimodal sentiment analysis: adaptive interaction and multi-scale fusion
Wang, H., Cao, J., Liu, J., & Song, J. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00957-1 · Cited by 6
“This study involves the following benchmark models:TFN (Zadeh et al., 2017), LMF (Liu et al., 2018), MULT (Tsai et al., 2019), MISA Hazarika et al. (2020), Self-MM (Yu et al., 2021), CubeMLP (Sun et al., 2022), COSMIC (Ghosal et al., 2020), MMIM (Han et al., 2021), BBFN (Han et al., 2021), MM-DFN (Hu et al., 2022a), M…”
InfluentialHe, J., Shi, X., Hu, C.-H., Mi, J., Li, X., & Toda, T. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3614428 · Cited by 7
“Hazarika et al. [10] proposed MISA, which disentangles each modality into invariant and specific subspaces through factorization, facilitating effective fusion for emotion prediction.”
Xia, P., Huang, Y., Qin, C., & Liu, C. (2026) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2026.114693 · Cited by 6
“As a result, strict synchronization of individual time-frequency frames is not required at the data processing stage.”
Detection Of Public Speaking Anxiety: A New Dataset And Algorithm
Song, W., Wu, B., Zheng, C., & Zhang, H. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00448 · Cited by 6
Multimodal Emotion Fusion Mechanism and Empathetic Responses in Companion Robots
Liu, X., Lv, Q., Li, J., Song, S., & Cangelosi, A. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3442203 · Cited by 6
“…73.40 MulT [61] V,A,T 83.3 82.9 FmlMSN [64] V,A,T 80.09 79.78 CubeMLP [68] V,A,T 85.6 85.5 EF-LSTM [65] V,A,T 78.48 78.51 MCTN [69] V,A,T 79.30 79.10 MISA [67] V,A,T 82.1 82.0 ICCN [66] V,A,T 83.0 83.0 BBFN [70] V,A,T 84.3 dataset by combining them with our facial expression recognition model.”
InfluentialLi, Y., Weng, W., Liu, C., & Lin, L. (2024) · Pattern Analysis and Applications · doi:10.1007/s10044-024-01320-w · Cited by 6
“To enable the model to learn the differences and consistency among modalities, Hazarika et al. [7] decomposed each modality into modality-invariant and modality-specific representations to learn the consistent and complementary information of each modality.”
DECNet: A Non-Contacting Dual-Modality Emotion Classification Network for Driver Health Monitoring
Dong, Z., Hu, C., Zhou, S., Zhu, L., Wang, J., Chen, Y., Lv, X., & Ji, X. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00042 · Cited by 6
Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Better
Ge, M., Li, M., Tang, D., Li, P., Liu, K., Deng, S., Pu, S., Liu, L., Song, Y., & Zhang, T. (2024) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3689092.3689415 · Cited by 9
“Among these approaches, multimodal technology with its capability to integrate different modal information sources for comprehensive analysis, has garnered significant attention[2][3].”
BackgroundManifold knowledge-guided feature fusion network for multimodal sentiment analysis
Wang, X., Wang, M., Cui, H. N., & Zhang, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127537 · Cited by 6
Wang, C., Zhang, Q., Dong, J., Fang, H., Schaefer, G., Liu, R., & Yi, P. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113638 · Cited by 6
CLIP-driven attention network for multimodal sentiment analysis
Lv, J., Yang, Q., Tian, S., Liu, B., & Yu, L. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07411-7 · Cited by 6
Gao, H., Cai, Z., Wang, X., Wu, M., & Liu, C. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3597398 · Cited by 8
“These settings follow configura-tions used in prior multimodal learning works [12], [22], [35], [36], [37].”
MODIFF-8 to better motivate: Live adaptive human-socially interactive agent interaction
Younsi, N., Pélachaud, C., & Chaby, L. (2026) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-026-00477-4 · Cited by 7
CM-CS: Cross-Modal Common-Specific Feature Learning For Audio-Visual Video Parsing
Chen, H., Zhu, D., Zhang, G., Shi, W., Zhang, X., & Li, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097072 · Cited by 6
Wang, Z., Nie, P., Liu, J., He, J., Wu, H., & Guo, P. (2024) · High-speed Railway · doi:10.1016/j.hspr.2024.04.003 · Cited by 6
Multimodal Sentiment Analysis for the Malay Language: New Corpus using CNN-based Framework
Taylor, S., & Fauzi, F. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3703445 · Cited by 5
MIMCL: Multilayer Interaction Module with Contrastive Learning for Speech Emotion Recognition
Li, F., Liu, R., & Wang, B. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733388 · Cited by 5
Self-attention mechanism prior to modality fusion for multimodal sentiment analysis
Chen, Z., Lu, C., & Wang, Y. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01868-5 · Cited by 5
Boosting Modal-Specific Representations for Sentiment Analysis With Incomplete Modalities
Jiang, X., He, L., Gao, F., Zhang, K., Li, J., & Gao, X. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3590909 · Cited by 5
“Multimodal sentiment analysis aims to integrate various modalities such as text, audio and visual to extract emotional information embedded within them [11], [32], [33], [34], [35], [36], [37].”
He, X., Pan, Y., Chen, Y., Li, Z., Xu, Z., Yang, C., & Wang, K. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3597474 · Cited by 5
“MISA [12] exemplifies this approach by utilizing a shared encoder along with three dedicated encoders to separate mode-invariant from mode-specific subspaces.”
Multimodal large language model enhancement network for multimodal sentiment analysis
Zhu, H., Shen, Z., Dai, C., & Yu, Z. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01914-2 · Cited by 5
Multimodal sentiment analysis based on temporal perception and cross-modal interaction
Ji, C., Tan, T., & Dai, W. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01954-8 · Cited by 5
Gradient and Structure Consistency in Multimodal Emotion Recognition
Shi, Q., Ye, M., Huang, W., Du, B., & Zong, X. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3608664 · Cited by 5
“To extract both modality-specific and shared information, recent approaches [10], [11], [12], [13], [37] have developed frameworks that separately model the unique and common aspects of each modality.”
InfluentialAffection-Guided Bottleneck Diffusion for Missing Modality Issue in Multimodal Affective Computing
Lin, R., He, Q., Zeng, Y., Mai, S., Huang, L., & Hu, H. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3632063 · Cited by 7
“Sincemultimodalfusionstrategies heavily depend on the existence of dominant modality [31], [36], [39], this implicit alignment is a key advantage of AB-Diff.”
Text-centric disentangled representation interaction network for Multimodal Sentiment Analysis
Hou, J., Omar, N., Tiun, S. B., Saad, S., & He, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130857 · Cited by 5
Uncertainty modeling in multimodal speech analysis across the psychosis spectrum
Rohanian, M., Hüppi, R. M., Nooralahzadeh, F., Dannecker, N., Pauli, Y., Surbeck, W., Sommer, I. E., Hinzen, W., Langer, N., Krauthammer, M., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02309-3 · Cited by 12
“To integrate the temporal dynamics of both audio and text modalities effectively, we adopted an uncertainty-aware fusion strategy inspired by the Calibrated and Ordinal Latent Distributions (COLD) Fusion method proposed by Hazarika et al. [60, 61].”
TPFN: A Text-Guided Progressive Fusion Network for Multimodal Sentiment Analysis
Ou, L., & Li, Z. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3663280 · Cited by 5
“In the unimodal tasks, the embedding di-mension of BERT is 768, with hidden layer dimensions of [64, 16, 32] for text, audio, and visual modalities, respectively.”
Cai, L., Liu, D., Liu, L., & Mou, R. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2026.3678350 · Cited by 7
“First, in feature representation learning, many earlier approaches [6], [7], [8], [9], [10], [11], [12], [13] have explored to extract temporal features from raw audio and video sequences.”
SGIR: Star Graph-Based Interaction for Efficient and Robust Multimodal Representation
Ding, Z., Lan, G., Song, Y., & Yang, Z. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3321404 · Cited by 5
“Following the previous works [24], [26], [41], we apply the preprocessing and unimodal encoding of raw features for different datasets.”
BackgroundMethodInfluentialSAAML: A Framework for Semi-supervised Affective Adaptation via Metric Learning
Tran, M., Kim, Y., Su, C.-C., Kuo, C.-H., & Soleymani, M. (2023) · ACM Multimedia · doi:10.1145/3581783.3612286 · Cited by 6
Xu, G., Deng, L., Li, Y., Wei, Y., Zhou, X., & Zhu, H. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3324462 · Cited by 5
“Hazarika et al. [19] proposed to design two encoders that could map the representation of each modal into the modality-specific and modality-invariant spaces, respectively.”
BackgroundMultimodal Sentiment Analysis Based on 3D Stereoscopic Attention
Huang, J., Pu, Y., Zhou, D., Shi, H., Zhao, Z., Xu, D., & Cao, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446572 · Cited by 5
Hwang, Y., & Kim, J.-H. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00557 · Cited by 5
Multimodal Sentiment Analysis Using Deep Learning: A Review
Patel, S., Shroff, N., & Shah, H. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-59097-9_2 · Cited by 5
Chen, X., Shuai, Q., Hu, F., & Cheng, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128794 · Cited by 7
Emoji multimodal microblog sentiment analysis based on mutual attention mechanism
Lou, Y., Zhou, J., Zhou, J., Ji, D., & Zhang, Q. (2024) · Scientific Reports · doi:10.1038/s41598-024-80167-x · Cited by 5
A dynamic weighted fusion model for multimodal sentiment analysis
Yang, L. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-04201-9 · Cited by 5
Metadata
- Authors
- Hazarika, D., Zimmermann, R., & Poria, S.
- Affiliations
- National University of Singapore (SG)
- Singapore University of Technology and Design (SG)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 1122-1131
- OpenAlex
- W3093051361
- Cited by
- 1,073
- References
- 45
- Citations by year
- 2026: 280 · 2025: 315 · 2024: 238 · 2023: 136 · 2022: 78 · 2021: 23 · 2020: 3
- FWCI
- 22.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- Funders
- Ministry of Education - Singapore (MOE2018-T2-1-103)
- DSO National Laboratories - Singapore (RTDST190702)
- RIE 2020 Advanced Manufacturing and Engineering (AME) programmatic grant (A19E2b0098)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, modality gap, modality-invariant representations, modality-specific representations, subspace projection, multimodal fusion, user-generated video, humor detection, MOSI, MOSEI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
