The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Open accessAbstract
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On CMU-MOSEI dataset, our proposed C-MIB yields 0.7% improvement on Acc2 and 1.2% improvement on F1 score compared to the current state-of-the-art MISA [53]. The performance on Acc7 and Corr is also improved compared to the state-of-the-art methods.”
From Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations (2023) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 8×
“Inter-utterance context. et al. proposed one of the first models, bc-LSTM , which utilized”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“However, following recent works [7], including the stateof-the-art ICCN [50], we utilize the pre-trained BERT [11] as the feature extractor for textual utterances.”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“ndard low-level features that are provided by the respective benchmarks and utilized by the state-of-the-art methods. 4.3.1 Language Features. Traditionally, language modality features has been GloVe [38] embeddings for each token in the utterance. However, following recent works [7], including the stateof-the-art ICC…”
MethodInfluential(2008) · Journal of Machine Learning Research · Cited by 33,051
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 5×
“licly available models of [26, 51, 58]).”
BackgroundInfluentialSelf-citationEkman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,971 · cited 1×
“Both MOSI and MOSEI use Facet 3 to extract facial expression features, which include facial action units and face pose based on the Facial Action Coding System (FACS) [14].”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · cited 1×
“For UR_FUNNY, OpenFace [4], a facial behavioral analysis tool, is used to extract features related to the facial expressions of the speaker.”
Method
Research works printed but not linked: 15
- Andrew Galen, 2013, Proceedings of the 30th International Conference on Machine Learning, ICML 2013 (JMLR Workshop and Conference Proceedings, 1255
- Bousmalis Konstantinos, Advances in Neural Information Processing Systems 29. Curran Associates
- 10.5555/2354409.2355082 · DOI
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 19
- Drugman Thomas, 2011, Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonics. In INTERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association. ISCA
- Kiela Douwe, 2019, NeurIPS 2019 Workshop. Curran Associates, Inc.
- Lu Jiasen, Advances in Neural Information Processing Systems 32. Curran Associates
- 10.5555/2392963.2392965 · DOI
- Peng Minlong, 2018, Long Papers
- Rajagopalan Shyam Sundar, 2016, Proceedings, Part VII (Lecture Notes in Computer Science, 353
- Salzmann Mathieu, 2010, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010 (JMLR Proceedings, 708
- Hubert Tsai Yao-Hung, 2019, Learning Factorized Multimodal Representations. In 7th International Conference on Learning Representations, ICLR
- Vaswani Ashish, Advances in Neural Information Processing Systems 30. Curran Associates
- Zadeh Amir, 2018, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, 2246
- Zellinger Werner, 2017, Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning. In 5th International Conference on Learning Representations, ICLR
Cited by
Sorted by citations
Items 501–600 of 1,001
SMURF: Statistical Modality Uniqueness and Redundancy Factorization
Wörtwein, T., Allen, N. B., Cohn, J. F., & Morency, L. (2024) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3678957.3685716 · Cited by 3
Xia, H., Chen, J., Tan, Y., & Tang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8502-5_30 · Cited by 2
$\mathcal {P}$owMix: A Versatile Regularizer for Multimodal Sentiment Analysis
Georgiou, E., Avrithis, Y., & Potamianos, A. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3496316 · Cited by 3
“Hazarika et al. [10] augment the learning objective with feature reconstruction loss as well as attracting and repelling objectives and Yang et al. [42] learn a common space as well as a private space for each of the involved modalities.”
InfluentialZhu, C., Chen, M., Li, H., Zhang, S., Liang, H., Sun, C., Liu, Y., & Chen, J. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10678-1 · Cited by 2
“All of the compared models are divided into two groups: (1) representation learning-based approaches: MISA [9], Self-MM To measure model performance using essential classification metrics, the regression interval for MOSI/ MOSEI and SIMS/SIMSV2(s) is subdivided into seven and five sentiment…”
Using EEG technology to enhance performance measurement in physical education
Zhai, Z., Han, L., & Zhang, W. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1551374 · Cited by 2
Liu, X., Chen, Q., Cheung, Y., & Peng, S. (2025) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2025.3550645 · Cited by 2
“Similar to work [29], we extract 35-dimensional visual features, primarily comprising basic and advanced facial action units, from video frames using FACET [30].”
InfluentialYan, Z., Zhang, Y., Deng, J., & Ren, F. (2025) · Research · doi:10.34133/research.0729 · Cited by 2
“In this section, we investigate the effect of selecting varying numbers of local representations from the audio and video MISA(BERT) [ 27 ] 69.62 66.18 MISA(ALBERT) [ 27 ] 69.82 66.18 MAG(ALBERT) [ 22 ] 67.20 69.12 MAG(XLNet) [ 22 ] 72.43 76.47 AGM (BERT) [ 26 ] 65.97 - FDMER (BERT) [ 28 ] 71.87 -…”
InfluentialAdvanced Modality Contribution-based Fusion for Multi-modal Multi-label Emotion Recognition
Nie, J., Dong, W., & Mao, Q. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-04212-6 · Cited by 2
Guo, S., Chen, T., Wang, P., Yan, J., & Liu, H. (2025) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2025.3576777 · Cited by 2
“To address this issue, multimodal target recognition technology that integrates domain knowledge [6], [7], [8] offers a novel solution.”
Wen, Q., Wei, P., Li, F., Hu, Q., Zeng, B., & Feng, G. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733355 · Cited by 2
“Following previous works [8, 28, 37], we utilize the powerful BERT Encoder [13] to encode the raw text tokens 𝑋 into contextual word embeddings.”
Hierarchical Signal Calibration and Refinement for Multimodal Sentiment Analysis
Ren, B., Cao, T., Zhang, Z., Bai, S., & Liu, N. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3603884 · Cited by 2
“Disentangled representation-based methods: MISA [30], MMCL [33], TMBL [36].”
Jin, Y., Wang, Y., Wang, Y., Chen, Q., Hu, B., Han, Y., Ma, C., & Pedrycz, W. (2025) · International Journal of Information Systems in the Service Sector · doi:10.4018/ijisss.388002 · Cited by 2
Mou, L., Zhen, S., Mao, S., & Ma, N. (2025) · doi:10.1145/3746277.3760407 · Cited by 2
D$$^2$$M: Decomposition-diffusion model for enhanced multimodal sentiment analysis
Zong, L., Xu, W., Hu, D., & Wang, J. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06979-0 · Cited by 2
Step-Wise Prompting Meets Uncertainty-Aware Dynamic Fusion for Robust EEG-Visual Emotion Recognition
Hao, D., Meng, M., Gao, Y., Lou, X., & Kong, W. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3632304 · Cited by 2
“These findings align with established multimodal learning theories emphasizing the need to balance shared and unique information [12], [66], [67], [68].”
FADD-Net:Multimodal Sentiment Analysis Based on Feature Alignment and Differential Decoupling
Zhang, J., Zhang, S., Liu, J., Wen, H., Zhang, Y., & Chen, M. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228674 · Cited by 2
Attention-Based Denoising Cross-Modal Multi-Level Fusion Network for Multimodal Sentiment Analysis
Wang, K., & Pan, S. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227992 · Cited by 2
“Furthermore, traditional fusion methods, which typically concatenate multimodal features and feed them into an MLP or fully connected layer [12], [13], fail to fully exploit complementary information between modalities and overlook potential higher-order correlations.”
From Feature Alignment to Multimodal Fusion: A Two-Stage Primary Modality-Guided Approach for MSA
Ma, G., Ren, X., Jiang, Y., Guan, H., & Xu, B. (2025) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3743093.3771041 · Cited by 2
Visual Semantic Knowledge Discovery for Multimodal Intent Recognition
Cheng, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3642324 · Cited by 2
“MISA [8] captures modality-invariant and modality-specific representations from each modality and then concatenates them with a self-attention mechanism for fine-grained fusion.”
A text guided multimodal scale path fusion network for multimodal sentiment analysis
Liu, S., Zhao, H., Chen, Y., Kong, F., & Li, K. (2025) · Scientific Reports · doi:10.1038/s41598-025-32637-z · Cited by 2
“Hazarika et al. advanced this paradigm in their MISA framework 28 , decoupling modality-invariant and modality-specific representations while enforcing cross-granular semantic consistency through multi-scale contrastive learning objectives.”
Frequency-aware experts with multi-stage fusion for multimodal sentiment analysis
Zhu, X., & Li, Y. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-01006-7 · Cited by 4
“Early symmetric methods (Hazarika et al., 2020) treat all modalities equally, overlooking modality dominance and often diluting strong cues such as language.”
LUCID: lexicon-augmented fusion with ordinal calibration for multimodal sentiment analysis
Yuan, F., Qi, K., Liu, J., Lu, C., & Chen, W. (2026) · Multimedia Systems · doi:10.1007/s00530-025-02208-3 · Cited by 2
TB-MEN: Text-Centric Bidirectional Modality Enhancement Network for Multimodal Sentiment Analysis
Zhang, J., Yin, M., Wei, G., Shi, D., & Guo, Y. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3660469 · Cited by 2
“(cid:2) Modality-Invariant and − Specific Representations for Multimodal Sentiment Analysis (MISA) [14].”
Xie, Y. (2026) · International Journal on Semantic Web and Information Systems · doi:10.4018/ijswis.402041 · Cited by 2
Yu, J., Li, M., Deng, H., Gao, W. G. W., & Zhu, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7081-2_34 · Cited by 2
Beneficial Noise Learning for Robust Multimodal Fusion
Zhu, A., Hu, M. Q., Wang, X., & Ren, F. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3668492 · Cited by 3
“To assess BRMF, we compare it with a broad range of multimodal fusion methods, covering classical models (e.g., TFN [6], LMF [23], MISA [29], EFT [14], LFT [14], MAG-BERT [24],), recent state-of-the-art approaches (e.g., TMBL [51], ConKI [28], MulT [25], ALMT [15]), and several emerging techniques…”
CMD$^{3}$: Cross-Modal Decoupled Deformable Distillation for EEG-fNIRS Fusion
Fan, T., Tian, F., Wang, S., Zhang, H., Luo, G., Zhu, L., Liu, J., Jiang, L., Cai, R., Dong, Q., et al. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3670503 · Cited by 2
“Therefore, recent studies have explored more effective fusion strategies in both modality-independent and modality-specific feature spaces [40], [41], [42].”
Task-Aware Information Decoupling for Multimodal Clustering
Jin, Z., Zheng, X., Tang, C., Li, C., Liu, Y., & Liu, X. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3679087 · Cited by 2
“Contemporary methods attempt to impose regularization constraints for retaining discriminative attributes, yet predominantly adopt simplistic strategies aimed at maximizing feature discrepancies across views [3], [31], [32].”
Liu, Z., Liu, S., Yan, W., Song, P., Song, Y., & Gao, R. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792577 · Cited by 2
Kushwaha, N., Cambria, E., & Hussain, A. (2026) · Cognitive Computation · doi:10.1007/s12559-026-10573-y · Cited by 2
“Cross-modal attention networks with uncertainty quantification have provided confidence estimates alongside predictions [11], addressing the need for reliable outputs in clinical applications.”
Multi-View Urban Region Embedding via Commonality-Specificity Disentanglement
Li, Z., Jia, H., Zhao, K., Huang, W., & Chen, M. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770854.3780234 · Cited by 2
Wang, B., Zhang, X., Du, K., Gao, C., & Li, L. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00117 · Cited by 2
A Dual-branch Enhanced Multi-task Learning Network for Multimodal Sentiment Analysis
Geng, W., Li, X., & Bian, Y. (2023) · International Conference on Multimedia Retrieval · doi:10.1145/3591106.3592260 · Cited by 2
“Table 3: Results on CMU-MOSEI; 1 is from [4]; 2 is from [21]; 3 is from [3]; Models with ∗ are reproduced from open-source”
BackgroundRecent Advancements and Challenges in Multimodal Sentiment Analysis: A Survey
Dong, Z.-X., & Liu, H. (2023) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc58545.2023.10327944 · Cited by 2
“Hazarika et al. [10] proposed Modality-Invariant and -Specific Represen-tations (MISA) framework, which maps each modality into two distinct feature spaces, namely, a modality-invariant feature space, in which the commonalities between modalities are represented, and a modality-specific feature…”
BackgroundResultAVMFAC: Acoustic Visual Multi-scale Feature Aggregation Conformer for Multimodal Sentiment Analysis
Zheng, Q. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8565-4_23 · Cited by 2
Rong, H., Chen, Z., Lu, Z., Xu, F., & Sheng, V. S. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3651983 · Cited by 2
“Similarly, Hao et al. [7] and Hazarika et al. [8] have extracted features from different modalities and linked the distinct modality features by Cross-Transformer achieving a fine-grained fusion of text and image representations.”
BackgroundSemantic-specific multimodal relation learning for sentiment analysis
Wu, R., Luo, Y., Liu, J., & Tang, X. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09644-8 · Cited by 3
Multimodal semantic enhanced representation network for micro-video event detection
Li, Y., Liu, X., Zhang, L., Tian, H., & Jing, P. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112255 · Cited by 2
Visual audio and textual triplet fusion network for multi-modal sentiment analysis
Lv, C., Zhang, X., & Zhang, H. (2024) · Signal Image and Video Processing · doi:10.1007/s11760-024-03563-w · Cited by 2
Enhancing Multimodal Sentiment Recognition Based on Cross-Modal Contrastive Learning
Cui, L., & Yang, S. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10688113 · Cited by 2
“Hazarika et al. address it by mapping features to modality-specific and invariant subspaces [5].”
BackgroundMethodA dissimilarity feature-driven decomposition network for multimodal sentiment analysis
Liu, M., & Li, Z. (2025) · Multimedia Systems · doi:10.1007/s00530-024-01660-x · Cited by 2
Decoupled Feature and Self-Knowledge Distillation for Speech Emotion Recognition
Yu, H., & Ning, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3542948 · Cited by 2
“Other studies, such as [24], [25], and [26], decoupled information from different modalities into modality-invariant and modality-specific features.”
Adversarial Training and Cross-modal Feature Fusion in Multimodal Sentiment Analysis
Li, J., Lin, C., Wang, H., Zhi, Y., Chen, J., & Huang, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890023 · Cited by 2
“In multi-modal sentiment recognition, each modality should remain consistent while providing complementary information [10] [11].”
Li, C., Xie, L., Wang, X., Pan, H., & Wang, Z. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01758-w · Cited by 2
BiMSA: Multimodal Sentiment Analysis Based on BiGRU and Bidirectional Interactive Attention
Wang, Q., Yu, H., Wang, Y., & Bian, H. (2025) · The European Journal on Artificial Intelligence · doi:10.1177/30504554251319444 · Cited by 2
“Additionally, Hazarika et al. (2020) proposed MISA, which projects each modality into two subspaces.”
Cluster-Granularity Spatiotemporal Transfer for Cross-Region Graph-Based Traffic Forecasting
Guo, C., Hwang, F.-J., Chen, C., Chang, C., & Chang, C. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3552111 · Cited by 3
“…illustrates the graph selector where the central moment discrepancy (CMD) [25] is used to evaluate the distribution discrepancy between the source and target data as it is an advanced indicator for evaluating the discrepancy between distributions by matching the order-wise moment differences [26].”
A Multimodal Emotion Recognition Model with Intra-Modal Enhancement and Inter-Modal Interaction
Zhang, Z., Wang, Y., Cui, J., & Zheng, H. (2025) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2024edp7161 · Cited by 2
FGMFN: Fine-Grained Multiscale Cross-Modal Sentiment Analysis in Advertisements
Wang, H., Chen, P., & Du, X. (2025) · IEEE Access · doi:10.1109/access.2025.3571624 · Cited by 2
“Additionally, the auxiliary training loss hyperparameters were inspired by MISA [48], and a grid search was conducted in Section IV-F to determine the optimal values for α and β .”
CU-SEMLP: All-MLP-based multimodal interaction model for multimodal sentiment analysis
Liu, S., Zhao, H., Chen, Y., Kong, F., & Li, K. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07364-x · Cited by 2
Feng, C., Yang, H., Wang, S., & Li, X. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07474-6 · Cited by 3
“The Modality Invariant and Specific Representations (MISA) [30] attempts to map the same feature into two different feature spaces: modality-invariant features and modality-specific features, and the author believes that modality-invariant and modality-specific features provide the basis for…”
Heterogeneous Graph Embedding for Multimodal Multi-Label Emotion Recognition
Hu, D., Jiang, X., Sun, Z., Shen, F., & Xu, X. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733353 · Cited by 4
Liu, Y., & Gao, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3584551 · Cited by 2
“To address these shortcomings, attention-based fusion models—such as the Multimodal Transformer—have been proposed to dynamically reweight each modality according to its relevance to the prediction task [25], [26].”
Multiscale Spherical Feature Decoupling Network for Multimodal Image Registration
Li, T., Zhou, Z., Zhu, Q. X., Luo, J., & Wang, Y. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3587800 · Cited by 2
“Hazarika et al. [56] proposed a multimodal affective framework that disentangles modalities into modality-shared and modality-unique features, integrating them to predict affective states.”
InfluentialA Review of Emotion-Aware Voice Systems in Sustainable Human–Machine Interaction
Tarunima, V. S., Harish, R., Dhanaselvam, J., & Singh, A. B. (2025) · International Conference on Sustainable Computing and Data Communication Systems (ICSCDS) · doi:10.1109/icscds65426.2025.11167087 · Cited by 2
“Each dataset presents unique challenges in terms of emotional granularity, speaker diversity, and modality coverage [33].”
Multimodal Humor Detection Based on Cross-Modal Attention and Modal Maximum Correlation
Quan, Z., Sun, T., Su, M., & Wei, J. (2022) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa54385.2022.10032426 · Cited by 2
“First, we obtained encouraging results compared to both the complex fusion mechanisms TFN [3], LMF [4], and the advanced representation learning model MISA [5], see (a) and (b) in “Fig.”
ResultSelf-supervised Multi-view Disentanglement for Expansion of Visual Collections
Jain, N., Vaddamanu, P., Maheshwari, P., Vinay, V., & Kulkarni, K. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3570425 · Cited by 2
FEENN: The Feature Enhancement Embedded Neural Network for Robust Multimodal Emotion Recognition
Wang, C., Dong, J., Sui, Y., Xu, P., Wu, Y. F., & Xu, Y. (2023) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3607865.3613180 · Cited by 2
PMFNet: A Progressive Multichannel Fusion Network for Multimodal Sentiment Analysis
Li, J., Tao, C., & Guan, D. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-8181-6_21 · Cited by 2
Zhao, T. G., Peng, J., Wang, L., & Zheng, C. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10393942 · Cited by 2
“• MISA: MISA [21] decomposes modalities into different modal feature subspaces and combines various losses, such as similarity loss and reconstruction loss, to learn modality-invariant and modality-specificity representations.”
Analyzing the Domain Robustness of Pretrained Language Models, Layer by Layer
Kashyap, A. R., Mehnaz, L., Malik, B., Waheed, A., Hazarika, D., Kan, M., & Shah, R. R. (2021) · National University of Singapore · Cited by 2
Self-citationLearning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal RetrievalPreprint
Zeng, D.-H., Wu, J., Hattori, G., Yu, Y., & Xu, R. (2021) · arXiv.org · doi:10.48550/arxiv.2110.13556 · Cited by 6
“In recent years, some works have applied audio-visual joint embedding learning to some tasks, such as Multimodal Sentiment Analysis [19], audio-visual separation [31], Visual2sound generation [73], and audio-visual cross-modal retrieval [66].”
BackgroundA Mixture-of-Experts Model for Multimodal Emotion Recognition in ConversationsPreprint
Dutta, S., Balaji, S., & Ganapathy, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5496422 · Cited by 2
Cross-modal Multiple Granularity Interactive Fusion Network for Long Document Classification
Liu, T., Hu, Y., Gao, J., Sun, Y., & Yin, B. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3631711 · Cited by 2
Research on the Application of Video Affective Computing in the Field of Radio and Television
Liu, W. (2023) · International Conference on Culture-oriented Science & Technology (ICCST) · doi:10.1109/cost60524.2023.00050 · Cited by 1
Multimodal Sentiment Analysis for Movie Scenes Based on a Few-Shot Learning Approach
Yang, H., & Li, B. (2024) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001424510091 · Cited by 1
Guo, Y., Qin, Y., Qin, X., Ren, Z., & Chen, L. (2023) · International Conference on Sensing, Measurement & Data Analytics in the era of Artificial Intelligence (ICSMD) · doi:10.1109/icsmd60522.2023.10490667 · Cited by 2
Greedy Fusion Oriented Representations for Multimodal Sentiment Analysis
Xu, R. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8620-6_39 · Cited by 1
Transferring Personality Knowledge to Multimodal Sentiment Analysis
Zhu, K., Liu, X., Xie, H., Cai, C., Fu, R., Li, G., Wen, Z., Tao, J., Fan, C., Lv, Z., et al. (2024) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp63861.2024.10800671 · Cited by 1
Liu, T., Hu, Y., Li, M., Yi, J., Chang, X., Gao, J., & Yin, B. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3537759 · Cited by 3
“MISA [21]: Integrating both modality-invariant and modality-specific features to forecast emotional states.”
APOLLO: Differential Private Online Multi-Sensor Data Prediction with Certified Performance
Xu, H., Li, W., Wu, S., Zhao, L., & Cai, Z. (2024) · Proceedings · doi:10.1109/icdm59182.2024.00060 · Cited by 1
“2) Baseline: MISA [26], Self-MM [27], and MMIM [28] represent the current state-of-the-art models on both the MOSI and MOSEI datasets for multi-sensor sentiment analysis.”
Multimodal Sentiment Analysis Based on Expert Mixing of Subtask Representations
Lei, L., He, W., Zheng, Q., & Zhu, B. (2025) · IEEE Access · doi:10.1109/access.2025.3549280 · Cited by 1
TDMER: A Task-Driven Method for Multimodal Emotion Recognition
Xu, Q., Gu, Y., Li, C., Zhang, H., Lin, H. X., & Liu, L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889666 · Cited by 1
“In cross-modal learning, Hazarika et al. [21] use attention mechanisms to push emotional information exchange between decoupled features.”
InfluentialSentiment Analysis for Live Video Comments with Diffused Fusion of Cross-modal Representations
Luo, C., Fang, L., & Qiu, B. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890276 · Cited by 1
Fine-grained Semantic Disentanglement Network for Multimodal Sarcasm Analysis
Tang, J., Ni, B., Zhou, F., Liu, D., Ding, Y., Peng, Y. K., Cichocki, A. S., Zhao, Q., & Kong, W. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3722558 · Cited by 1
“…Model [37], ICCN (Interaction Canonical Correlation Network) [32], MulT [36], MISA (modality-invariant and-specific representations-based model) [14], MAG-Bert [30], Self-MM (self-supervised multi-task multimodal sentiment analysis network) [43], MMIM (Multimodal Mutual Information…”
Improving Multimodal Sentiment Analysis with Unimodal Pseudo-label Generation
Zhao, J., Deng, O., & Jin, Q. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-86193-2_11 · Cited by 1
Wang, Y., Zhang, W., & Sun, N. (2024) · IEEE International Conference on Communication Technology (ICCT) · doi:10.1109/icct62411.2024.10946552 · Cited by 1
Wei, S. (2025) · IEEE Access · doi:10.1109/access.2025.3561179 · Cited by 1
“Pre-trained models, fine-tuned for specific applications, have been instrumental in decoding complex emotional states such as frustration, motivation, or relaxation [17].”
Guo, K., Tang, H., Xu, T., Zhou, H., Lyu, M., Liu, Z., Wang, X., & Li, X. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3569841 · Cited by 1
“The transformer encoder has emerged as a popular technique for modality fusion [7], [31], [32], [33], as it can combine information from multiple sources to learn a unified representation.”
Personality-Guided Multimodal Sentiment Analysis
Zhu, K., Liu, X., Xie, H., Cai, C., Fu, R., Yi, J., Tao, J., Wen, Z., Fan, C., & Lv, Z. (2024) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait63902.2024.11022007 · Cited by 1
Multi-Dilated Transformer for Multimodal Sentiment Analysis
Xie, S., Hu, Z., Xie, X., Mao, Y., Zhen, X., & Zeng, X. (2025) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit65432.2025.11035582 · Cited by 1
Adaptive Agent Semantic Aggregation Network for Multimodal Sentiment Analysis
Su, Y., & Zhao, X. Y. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733266 · Cited by 1
Exploring Feature Interactions for Multimodal Sentiment Analysis
Xu, D., & Zhang, C. (2025) · International Journal of Cognitive Informatics and Natural Intelligence · doi:10.4018/ijcini.383754 · Cited by 1
Gong, X., Xue, Z., Liu, T., Sun, Y., & Zhang, Y. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01895-2 · Cited by 1
Multimodal Rumor Detection by Online Balance Multimodal Representation Learning
Ren, J., & Zhong, T. (2025) · IEEE Access · doi:10.1109/access.2025.3587007 · Cited by 1
“The first strategy addresses architectural limitations by designing modality-specific encoders that capture unique features alongside modality-invariant encoders that extract shared representations [7].”
Incorporating Communication Style and Interaction of Speakers for Sarcasm Explanation in Dialogue
Li, Y., Zhang, W., Lin, Z., Ding, G., & Wang, W. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730006 · Cited by 1
Multimodal sentiment analysis based on slice aggregation and dynamic fusion
Zhan, Z., Cao, D., Chen, Z., Cheng, H., & Yu, Z. (2025) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-025-00195-y · Cited by 2
“Hazarika et al. (2020) suggested a sentiment analysis framework by projecting each modality into modality-invariant and modality-specific subspaces, which can capture the common representations across modalities and the specific representations for each modality.”
A Character-Image-Modality and Multi-Label Auxiliary Model for Chinese Sentiment Analysis
Wang, L., Chen, Z., Zhang, M., & Palade, V. (2025) · IEEE Access · doi:10.1109/access.2025.3598452 · Cited by 1
Altering Query Prompting With Contrastive Learning for Multimodal Intent Recognition
Jia, Y., Wang, X., Shao, Z., & Liu, M. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3599107 · Cited by 1
“In this subsection, we comprehensively evaluate our proposed AQP-CL method against multiple state-of-the-art approaches, including text-only baselines (TEXT) and multimodal frameworks such as MAG-BERT [21], MISA [22], MulT [10], MMIM [23], SPECTRA [24], TCL-MAP [5], CAGC [6] and MIntOOD [9].”
MSAF: Multimodal Sentiment Detection via Multiscale Adaptive Fusion
Guan, J., Shu, Y., Liu, W., Li, W., Zhou, S., & Zhang, Y. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3602409 · Cited by 1
“Hazarika et al. [41] introduced a modality-invariant model called MISA, which addresses the modality alignment issue by dividing each modality into common and unique feature subspaces.”
Chang, H.-Y., Liu, A.-S., Lin, Y., Liu, C., Lee, L.-E., Chen, F., Hung, S., & Fu, L. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3604933 · Cited by 1
“Inspired by the experimental results in previous works [3], [5], [9], [10], [11], [12], [13], [14], [29], [30], we consider the text modality as the dominant modality and employ acoustic and visual representations to reinforce the text modality through a specially designed Modality Reinforcement…”
InfluentialLiu, C., Wang, S., Liu, Q., Wang, Y., & Wang, F. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3606949 · Cited by 1
“The backbone networks used were either Simple Fusion Network (SFN) or the classic MISA [49].”
Hierarchical prompt fusion and image denoising for multimodal aspect-based sentiment analysis
Fuxian, Z., & Xu, X. (2025) · Memetic Computing · doi:10.1007/s12293-025-00475-1 · Cited by 1
Murthy, J. S., Kandaswamy, D., & Lai, W. (2025) · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss65446.2025.11149904 · Cited by 1
“Similarly, BN-WVAD [14] and RTFM [15] employ multi-instance learning to refine anomaly classification, while MISA [16], MM-ViT [17], and MERLOT [18] introduce multimodal and transformer-based architectures.”
He, Z., & Wang, Y. (2025) · Biomedical Physics & Engineering Express · doi:10.1088/2057-1976/ae09b2 · Cited by 1
Multi-Task Hypergraph-Attention Framework for Multimodal Sentiment Analysis
Wang, Y., Yang, Z., Wang, L., Liu, M., & Chen, Y. (2025) · IEEE Vehicular Technology Conference · doi:10.1109/vtc2025-spring65109.2025.11174446 · Cited by 1
Wei, Y., & Chen, J. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07930-3 · Cited by 2
“Next, we assessed models that emphasize modality consistency and differences, such as MISA [16], Self-MM [60], MAG-BERT [41], and MMIM [15].”
A Lightweight Two-Stage Attention-Gating Network for Efficient Multimodal Sentiment Analysis
Wang, Y., Xie, W., Yang, Z., Xu, C., Lin, Y., & Chen, Y. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3626046 · Cited by 1
“However, to achieve a more nuanced fusion, subsequent models like MISA [11] have focused on disentangling representations into modality-specific (private) and modality-invariant (shared) components.”
InfluentialAmd'SaEr: Arabic Multimodal Dataset for Sentiment Analysis and Emotion Recognition
Haouhat, A., Bellaouar, S., Nehar, A., & Cherroun, H. (2025) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3774880 · Cited by 1
Feature fusion based transformer for sentiment analysis in social networks
Li, S., Li, H., Du, J., Yan, S., & Dong, C. (2025) · PLoS ONE · doi:10.1371/journal.pone.0333416 · Cited by 1
Multimodal Sentiment Analysis via Progressive Fusion of Audio-Visual Affective Descriptions
Ou, L., & Li, Z. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761386 · Cited by 1
Metadata
- Authors
- Hazarika, D., Zimmermann, R., & Poria, S.
- Affiliations
- National University of Singapore (SG)
- Singapore University of Technology and Design (SG)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 1122-1131
- OpenAlex
- W3093051361
- Cited by
- 1,073
- References
- 45
- Citations by year
- 2026: 280 · 2025: 315 · 2024: 238 · 2023: 136 · 2022: 78 · 2021: 23 · 2020: 3
- FWCI
- 22.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- Funders
- Ministry of Education - Singapore (MOE2018-T2-1-103)
- DSO National Laboratories - Singapore (RTDST190702)
- RIE 2020 Advanced Manufacturing and Engineering (AME) programmatic grant (A19E2b0098)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, modality gap, modality-invariant representations, modality-specific representations, subspace projection, multimodal fusion, user-generated video, humor detection, MOSI, MOSEI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
