The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Open accessAbstract
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On CMU-MOSEI dataset, our proposed C-MIB yields 0.7% improvement on Acc2 and 1.2% improvement on F1 score compared to the current state-of-the-art MISA [53]. The performance on Acc7 and Corr is also improved compared to the state-of-the-art methods.”
From Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations (2023) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 8×
“Inter-utterance context. et al. proposed one of the first models, bc-LSTM , which utilized”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“However, following recent works [7], including the stateof-the-art ICCN [50], we utilize the pre-trained BERT [11] as the feature extractor for textual utterances.”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“ndard low-level features that are provided by the respective benchmarks and utilized by the state-of-the-art methods. 4.3.1 Language Features. Traditionally, language modality features has been GloVe [38] embeddings for each token in the utterance. However, following recent works [7], including the stateof-the-art ICC…”
MethodInfluential(2008) · Journal of Machine Learning Research · Cited by 33,051
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 5×
“licly available models of [26, 51, 58]).”
BackgroundInfluentialSelf-citationEkman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,971 · cited 1×
“Both MOSI and MOSEI use Facet 3 to extract facial expression features, which include facial action units and face pose based on the Facial Action Coding System (FACS) [14].”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · cited 1×
“For UR_FUNNY, OpenFace [4], a facial behavioral analysis tool, is used to extract features related to the facial expressions of the speaker.”
Method
Research works printed but not linked: 15
- Andrew Galen, 2013, Proceedings of the 30th International Conference on Machine Learning, ICML 2013 (JMLR Workshop and Conference Proceedings, 1255
- Bousmalis Konstantinos, Advances in Neural Information Processing Systems 29. Curran Associates
- 10.5555/2354409.2355082 · DOI
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 19
- Drugman Thomas, 2011, Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonics. In INTERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association. ISCA
- Kiela Douwe, 2019, NeurIPS 2019 Workshop. Curran Associates, Inc.
- Lu Jiasen, Advances in Neural Information Processing Systems 32. Curran Associates
- 10.5555/2392963.2392965 · DOI
- Peng Minlong, 2018, Long Papers
- Rajagopalan Shyam Sundar, 2016, Proceedings, Part VII (Lecture Notes in Computer Science, 353
- Salzmann Mathieu, 2010, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010 (JMLR Proceedings, 708
- Hubert Tsai Yao-Hung, 2019, Learning Factorized Multimodal Representations. In 7th International Conference on Learning Representations, ICLR
- Vaswani Ashish, Advances in Neural Information Processing Systems 30. Curran Associates
- Zadeh Amir, 2018, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, 2246
- Zellinger Werner, 2017, Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning. In 5th International Conference on Learning Representations, ICLR
Cited by
Sorted by citations
Items 101–200 of 1,001
A feature-based restoration dynamic interaction network for multimodal sentiment analysis
Zeng, Y., Li, Z., Chen, Z., & Ma, H. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107335 · Cited by 42
VLP2MSA: Expanding vision-language pre-training to multimodal sentiment analysis
Yi, G., Fan, C., Zhu, K., Lv, Z., Liang, S., Wen, Z., Pei, G., Li, T., & Tao, J. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111136 · Cited by 43
EMOE: Modality-Specific Enhanced Dynamic Emotion Experts
Fang, Y., Huang, W., Wan, G., Su, K., & Ye, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01335 · Cited by 52
“On the other hand, to address modality specialization disappearance, related methodologies introduce various modality decoupling solutions to preserve modality-specific information [14, 29, 55], separating each modality into invariant and specific spaces before fusion.”
InfluentialRate-Adaptive Coding Mechanism for Semantic Communications With Multi-Modal Data
He, Y., Yu, G., & Cai, Y. (2023) · IEEE Transactions on Communications · doi:10.1109/tcomm.2023.3335977 · Cited by 46
Learning What and When to Drop
Chen, F., Sun, Z., Ouyang, D., Liu, X., & Shao, J. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475661 · Cited by 40
“More recently, researchers usually resort to a more advanced line of modal equilibrium exploration, where additional modalityspecific and modality-shared networks are used, to capture rich intra-modal and inter-modal relations [4, 12, 36], as shown in Figure 2(b).”
BackgroundMethodTriple disentangled representation learning for multimodal affective analysis
Zhou, Y., Liang, X., Chen, H., Zhao, Y., Chen, X., & Yu, L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102663 · Cited by 41
TF-BERT: Tensor-based fusion BERT for multimodal sentiment analysis
Hou, J., Omar, N., Tiun, S. B., Saad, S., & He, Q. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107222 · Cited by 43
Multimodal Reconstruct and Align Net for Missing Modality Problem in Sentiment Analysis
Luo, W., Xu, M., & Lai, H. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-27818-1_34 · Cited by 39
Zhang, Y., Yu, Y., Zhao, D., Li, Z., Wang, B., Hou, Y., Tiwari, P., & Qin, J. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3298328 · Cited by 43
“Hazarika et al. [29] proposed a framework, MISA, which projected each modality to two distinct subspaces.”
BackgroundFcdnet: Fuzzy Cognition-Based Dynamic Fusion Network for Multimodal Sentiment Analysis
Liu, S., Luo, Z., & Fu, W. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3407739 · Cited by 40
NHFNET: A Non-Homogeneous Fusion Network for Multimodal Sentiment Analysis
Fu, Z., Liu, F., Xu, Q., Qi, J., Fu, X., Zhou, A., & Li, Z. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859836 · Cited by 37
“To ensure fair competition with other baselines, we perform the same feature preprocessing for all three modalities following [19].”
MethodInfluentialToken-disentangling Mutual Transformer for multimodal emotion recognition
Yin, G., Liu, Y., Liu, T., Zhang, H., Fang, F., Tang, C., & Jiang, L. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108348 · Cited by 43
Li, Z., Huang, Z., Pan, Y., Yu, J., Liu, W., Chen, H., Luo, Y., Wu, D., & Wang, H. H. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124236 · Cited by 36
Disaster assessment from social media using multimodal deep learning
Shetty, N. P., Bijalwan, Y., Chaudhari, P., Shetty, J., & Muniyal, B. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19818-0 · Cited by 38
“This fusion method has been prominent in recent cross-modal learning approaches [28–33].”
Flexible Vertical Federated Learning With Heterogeneous Parties
Castiglia, T., Wang, S., & Patterson, S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3309701 · Cited by 42
“Centralized machine learning accommodates such multimodal data by supporting different local feature extractors and optimizers for each feature set [19], [20], [21], [22].”
Fine-Grained Disentangled Representation Learning For Multimodal Emotion Recognition
Sun, H., Zhao, S., Wang, X., Zeng, W., Chen, Y., & Qin, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447667 · Cited by 37
“…SMCN[19] 73.50 73.00 MCNN+BLSTM [6] 74.98 75.05 BERT + Wav2vec2 [20] - 76.31 Multi-granularity model [21] 74.80 75.50 SDTF-NET [22] 77.20 76.70 MISA [10] 76.18 75.73 Co-att + BAM+knowledge[23] 75.50 77.00 Ours 78.34 79.44 representations of different modalities without feature disentangle-ment.”
BackgroundInfluentialFDR-MSA: Enhancing multimodal sentiment analysis through feature disentanglement and reconstruction
Fu, Y., Huang, B., Wen, Y., & Zhang, P. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111965 · Cited by 35
MDKAT: Multimodal Decoupling With Knowledge Aggregation and Transfer for Video Emotion Recognition
Wang, J., Wang, C., Guo, L., Zhao, S., Wang, D., Zhang, S., Zhao, X., Yu, J., Wang, Y., Yang, Y., et al. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3571534 · Cited by 38
“MISA [23]: Modality-Invariant and -Specific Represen-tations for Multimodal Sentiment Analysis (MISA) learns both modality-independent and modality-specific features across modalities, improving multimodal sentiment analysis by reducing the modality gaps and capturing modality-specific characteristics.”
InfluentialGeneral Debiasing for Multimodal Sentiment Analysis
Sun, T., Ni, J., Wang, W., Jing, L., Wei, Y., & Nie, L. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612051 · Cited by 37
“Second, recent work [8] employs a more accurate formulation of negative/positive classes where negative and positive classes are assigned with < 0 and > 0 sentiment scores, respectively.”
BackgroundMethodInfluentialAdaMoW: Multimodal Sentiment Analysis Based on Adaptive Modality-Specific Weight Fusion Network
Zhang, J., Wu, X., & Huang, C. (2023) · IEEE Access · doi:10.1109/access.2023.3276932 · Cited by 36
“The sentiment prediction [37], [41], [45] and [9].”
BackgroundMethodAdapt and explore: Multimodal mixup for representation learning
Lin, R., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102216 · Cited by 34
EmoInt-Trans: A Multimodal Transformer for Identifying Emotions and Intents in Social Conversations
Singh, G. V., Firdaus, M., Ekbal, A., & Bhattacharyya, P. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3224287 · Cited by 36
Multimodal Graph for Unaligned Multimodal Sequence Analysis via Graph Convolution and Graph Pooling
Mai, S., Xing, S., He, J., Zeng, Y., & Hu, H. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3542927 · Cited by 63
“In our experiment, our model achieves a binary accuracy of 85.9% and a seven-way accuracy of 48.7% on CMU-MOSI dataset under the aligned setting, which suppresses the state-of-the-art models [17, 58].”
MethodInfluentialCohort-Individual Cooperative Learning for Multimodal Cancer Survival Analysis
Zhou, H., Zhou, F., & Chen, H. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3455931 · Cited by 39
“Previous multimodal methods [40], [41] simply decompose multimodal knowledge into common and specific components.”
Towards Robust Multimodal Sentiment Analysis Under Uncertain Signal Missing
Li, M., Yang, D., & Zhang, L. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3324552 · Cited by 33
“Most works [15], [16], [17], [18], [19] are based on a strong assumption that all modalities are complete during the training and testing phases.”
BackgroundEnsemble Pretrained Models for Multimodal Sentiment Analysis using Textual and Video Data Fusion
Liu, Z., Braytee, A., Anaissi, A., Zhang, G., Qin, L., & Akram, J. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3651971 · Cited by 37
“A comprehensive benchmarking was conducted to evaluate our models against MSA the recent state-of-arts methods, including LMF[16],TFN[32],MTAG [29],ICCN[23],MulT[25],andMISA[9].”
Wang, X., Lyu, J., Kim, B., Parameshachari, B. D., Li, K., & Li, Q. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3419140 · Cited by 38
Li, M., Zhu, Z., Li, K., & Pei, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3430045 · Cited by 33
“To address this challenge, researchers have designed a series of complex deep learning models [6], [7], [8], [9] aimed at learning rich and effective feature representations from every modality.”
Lei, Y., Yang, D., Li, M., Wang, S., Chen, J., & Zhang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-981-99-9119-8_18 · Cited by 35
A Unified Multimodal De- and Re-Coupling Framework for RGB-D Motion Recognition
Zhou, B., Wang, P., Wan, J., Liang, Y., & Wang, F. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3274783 · Cited by 38
“The NTU RGB-D is a large-scale human action dataset containing more than 56,000 multi-view RGB-D videos with 4 million frames that include 60 actions performed by 40 subjects.”
BackgroundMethodStructure Aware Multi-Graph Network for Multi-Modal Emotion Recognition in Conversations
Zhang, D., Chen, F., Chang, J., Chen, X., & Tian, Q. C. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3238314 · Cited by 33
“Previous works show that building modules individually for each modality helps capture their characteristic properties and features [33].”
BackgroundMultimodal Sentiment Analysis with Preferential Fusion and Distance-aware Contrastive Learning
Ma, F., Zhang, Y., & Sun, X. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00237 · Cited by 31
“Later, deep neural networks have been adopted to extract and fuse features for MSA [5]–[7].”
MethodMulti-modal news event detection with external knowledge
Lin, Z., Xie, J., & Li, Q. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103697 · Cited by 33
Cross-modal contrastive learning for multimodal sentiment recognition
Yang, S., Cui, L., Wang, L., & Wang, T. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05355-8 · Cited by 31
“MISA: In MISA [20], each modality was projected into two subspaces to create modality-invariant and specific representations: the first subspace represented modal invariant features, while the second subspace captured modal specific features.”
Noise-Resistant Multimodal Transformer for Emotion Recognition
Liu, Y., Zhang, H., Zhan, Y., Chen, Z., Yin, G., Wei, L., & Chen, Z. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02304-3 · Cited by 35
“For the MOSI and MOSEI datasets, referring to the existing methods (Tsai, Bai, et al., 2019; Hazarika et al., 2020; Mao et al., 2022), we used the features provided in the dataset, which had been extracted by the OpenFace (Baltrusaitis et al.”
InfluentialSentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errors
Wu, Y., Zhao, Y., Yang, H., Chen, S., Qin, B., Cao, X., & Zhao, W. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.109 · Cited by 33
Deng, H., Yang, Z., Hao, T., Li, Q., & Liu, W. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3211197 · Cited by 32
“…(Acc-7 or Acc-2), F 1 score, mean absolute error (MAE), and correlation (Corr) is adopted for CMU-MOSI and CMU-MOSEI by following the works [29], [35], [42]; Acc-2 and F 1 score are adopted for ICT-MMMO by following [34], [38], [39]; accuracy on three classes (Acc-3)and F 1…”
BackgroundMethodInfluentialAcFormer: An Aligned and Compact Transformer for Multimodal Sentiment Analysis
Zong, D., Ding, C., Li, B., Li, J., Zheng, K., & Zhou, Q. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611974 · Cited by 33
“Under word-aligned settings , we first consider We also utilize the pre-trained BERT-base-uncased [12] to extract [CLS] token [6, 19], we directly take the token-wise representations unaligned settings , we also extract 300-dimensional pre-trained Glove word embeddings and 768-dimensional…”
MethodUEFN: Efficient uncertainty estimation fusion network for reliable multimodal sentiment analysis
Wang, S., Ratnavelu, K., & Shibghatullah, A. S. (2024) · Applied Intelligence · doi:10.1007/s10489-024-06113-6 · Cited by 30
“4) The modality-invariant and -specific representations for MSA ( MISA ) [51] projects multimodal data into spaces of modality invariance and modality specificity.”
MISF-Net: Modality-Invariant and -Specific Fusion Network for RGB-T Crowd Counting
Mu, B., Shao, F., Xie, Z., Chen, H., Zhu, Z., & Jiang, Q. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3535330 · Cited by 31
“Inspired by the above facts and [34], our insight for the first key question ( RQ1 ) of RGB-T crowd counting is how to design an effective learning strategy to extract the modality-invariant and -specific feature [29]andBL+IADM[27]arethe generalmethods. from the two modalities to assist multi-modal…”
MAG+: An Extended Multimodal Adaptation Gate for Multimodal Sentiment Analysis
Zhao, X., Chen, Y., Li, W., Gao, L., & Tang, B. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746536 · Cited by 30
PIRNet: Personality-Enhanced Iterative Refinement Network for Emotion Recognition in Conversation
Lian, Z., Liu, B., & Tao, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3192469 · Cited by 33
“Modality factorization methods decompose multimodal features into distinct subspaces (such as MFM [70], MISA [71], and MMB2 [72]).”
MethodA transformer-encoder-based multimodal multi-attention fusion network for sentiment analysis
Liu, C., Wang, Y., & Yang, J. B. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05623-7 · Cited by 32
“To explore inter-modal differences, Hazarika et al. [38] modeled inter-modal similarity loss, difference loss, reconstruction loss, and task loss, respectively; Yu et al. [39] identified quantitative relationships between unimodal and multimodal sentiment and optimized them.”
Text-centered cross-sample fusion network for multimodal sentiment analysis
Huang, Q., Chen, J., Huang, C., Huang, X., & Wang, Y. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01421-w · Cited by 29
Wang, R., Yang, Q., Tian, S., Yu, L., He, X., & Wang, B. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129163 · Cited by 30
Multimodal Boosting: Addressing Noisy Modalities and Identifying Modality Contribution
Mai, S., Ya, S., Xiong, A., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3306489 · Cited by 34
“Following state-of-the-art methods [5], [77], [82], [84], we use BERT [57] to extract the high-level textual representation for multimodal sentiment analysis.”
BackgroundMethodPose-disentangled Contrastive Learning for Self-supervised Facial Representation
Liu, Y., Wang, W., Zhan, Y.-B., Feng, S., Liu, L.-Y. D., & Chen, Z. (2022) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.00937 · Cited by 43
“To achieve this, inspired by [2,22,34,46], the orthogonalizing regulation Lorth is defined as follows:”
BackgroundHe, J.-J., Shi, X.-H., Li, X.-F., & Toda, T. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp48485.2024.10446548 · Cited by 40
“In recent multimodal tasks like speech recognition [20] and sentiment analysis [21–23], researchers have proposed learning two distinct representations to enhance multimodal learning.”
BackgroundMBCFNet: A Multimodal Brain–Computer Fusion Network for human intention recognition
Li, Z., Zhang, G., Okada, S., Wang, L., Zhao, B., & Dang, J. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111826 · Cited by 29
Decoupled cross-attribute correlation network for multimodal sentiment analysis
Zhao, X., Li, X., Jiang, R., & Tang, B. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102897 · Cited by 31
GLoMo: Global-Local Modal Fusion for Multimodal Sentiment Analysis
Yan, Z., Zhang, Y., Hu, Z., Zhang, X., Deng, J., & Ren, F. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681527 · Cited by 33
Zeng, Y., Mai, S., & Hu, H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.109 · Cited by 31
Lin, R., & Hu, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.36 · Cited by 34
“MISA (Hazarika et al., 2020) Modality-Invariant and -Specific Representations projects modalities to two distinct subspaces, modality-invariant and -specific subspace, to provide a holistic view of multimodal data.”
BackgroundMethodInfluentialTowards Multimodal Sentiment Analysis Debiasing via Bias Purification
Yang, D., Li, M., Xiao, D., Liu, Y., Yang, K., Chen, Z., Wang, Y., Zhai, P., Li, K. J., & Zhang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73636-0_27 · Cited by 30
Multimodal Decoupled Distillation Graph Neural Network for Emotion Recognition in Conversation
Dai, Y., Li, Y., Chen, D., Li, J., & Lu, G. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3405406 · Cited by 28
“Restrictions apply. among modalities in each utterance [19], [35], [45], [46].”
BackgroundMethodA Multimodal Sentiment Analysis Method Based on Fuzzy Attention Fusion
Zhi, Y., Li, J., Wang, H., Chen, J., & Wei, W. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3434614 · Cited by 28
Text-Guided Reconstruction Network for Sentiment Analysis With Uncertain Missing Modalities
Shi, P., Hu, M. Q., Nakagawa, S., Zheng, X., Shi, X., & Ren, F. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3541743 · Cited by 29
“Modality-Invariant and -Specific Representations For MSA (MISA) [30]: This paper factorizes modalities into modality-invariant and modality-specific feature creativity.”
InfluentialQuan, Z., Sun, T., Su, M., & Wei, J. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/4767437 · Cited by 25
“*e former mainly uses neural networks such as the Long ShortTerm Memory (LSTM) Network [16] for the extraction of modal contextual information [10, 17].”
BackgroundMethodInfluentialDu, J., Jin, J., Zhuang, J., & Zhang, C. (2024) · Scientific Reports · doi:10.1038/s41598-024-54872-6 · Cited by 25
Dai, Y., Li, J., Li, Y., & Lu, G. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111954 · Cited by 27
Multimodal PEAR Chain-of-Thought Reasoning for Multimodal Sentiment Analysis
Li, Y., Lan, X., Chen, H., Lü, K., & Jiang, D. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3672398 · Cited by 29
“Then, employing the CoT reasoning process of LLMs to enhance the semantic representation of textual modality (Equation (3)) increased the accuracy to 86.0%.”
InfluentialLi, Z.-H., Liu, P.-B., Pan, Y.-S., Ding, W.-P., Yu, J., Chen, H.-R., Liu, W.-H., Luo, Y.-M., & Wang, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128940 · Cited by 25
Li, J., Wang, X., & Zeng, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3581236 · Cited by 25
“MERC is a sophisticated task that goes beyond the scope of traditional uni-modal [5], [6] and non-conversational [7], [8] affective endeavors (in this paper, the term “affective” encompasses emotional and sentimental aspects).”
CMJRT: Cross-Modal Joint Representation Transformer for Multimodal Sentiment Analysis
Xu, M. Y., Liang, F., Su, X., & Fang, C. (2022) · IEEE Access · doi:10.1109/access.2022.3219200 · Cited by 25
“Haz-arika et al. [20] propose MISA to learn modality-invariant and modality-specific representations to present a holistic perspective of multimodal data.”
BackgroundMethodLiu, R., Zhang, J., & Gao, G. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102257 · Cited by 26
Muti-Modal Emotion Recognition via Hierarchical Knowledge Distillation
Sun, T., Wei, Y., Ni, J., Liu, Z., Song, X., Wang, Y., & Nie, L. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3385180 · Cited by 26
“…validate our proposed model, we compared the model with several state-of-the-art baselines, including the feature-level fu-sionmodels( i.e.,EF-LSTM [43], TFN [22], MFM [16], MulT [3], MISA [44], LF-DNN [45], and ALBEF [20]) and decision-level fusion models ( i.e., FE2E [4], MESM [4], and UMT [46]).”
MethodDeep learning on multi-view sequential data: a survey
Xie, Z., Yang, Y. Z., Zhang, Y., Wang, J., & Du, S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10332-z · Cited by 28
Multimodal Reaction: Information Modulation for Cross-Modal Representation Learning
Zeng, Y., Mai, S., Yan, W., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3293335 · Cited by 25
“However, most attention-based methods like [45], [46] need to calculate the attention scores for each element and thus easily lead to higher computational costs.”
BackgroundMethodTACFN: Transformer-Based Adaptive Cross-Modal Fusion Network for Multimodal Emotion Recognition
Liu, F., Fu, Z., Wang, Y., & Zheng, Q. (2023) · CAAI Artificial Intelligence Research · doi:10.26599/air.2023.9150019 · Cited by 42
Huang, S., Qin, L., Wang, B., Tu, G., & Xu, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446922 · Cited by 25
“We apply transformer [16] as our deep interaction module, which has been shown highly effective in performing modalities fusion [17, 18, 19].”
MethodMulti-label Emotion Analysis in Conversation via Multimodal Knowledge Distillation
Anand, S., Devulapally, N. K., Bhattacharjee, S. D., & Yuan, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612517 · Cited by 23
Trujillo, J. P., & Holler, J. (2024) · Scientific Reports · doi:10.1038/s41598-024-52589-0 · Cited by 23
Crossmodal Translation Based Meta Weight Adaption for Robust Image-Text Sentiment Analysis
Zhang, B., Yuan, Z., Xu, H., & Gao, K. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3405662 · Cited by 27
“The first challenge is effectively detecting the modality-independent emotional behaviors in addition to the modality-invariant behaviors [11], [12], [13], [14].”
BackgroundMultimodal Sentiment Analysis With Mutual Information-Based Disentangled Representation Learning
Sun, H., Niu, Z., Wang, H., Yu, X., Liu, J., Chen, Y.-W., & Lin, L. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3529732 · Cited by 28
“In general, they qualitatively make invariant representations closer to each other and specific representations farther away from each other in geometric or distributional distance like [9], [10], [11].”
InfluentialHuang, C., Lin, Z., Huang, Q., Huang, X., Jiang, F., & Chen, J. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01806-y · Cited by 26
Actual Cause-Guided Adaptive Gradient Scaling for Balanced Multimodal Sentiment Analysis
Chen, J., Huang, Q., Huang, C., & Huang, X. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3736415 · Cited by 24
A Multitask Learning Framework for Multimodal Sentiment Analysis
Jiang, D., Wei, R., Liu, H., Wen, J., Tu, G., Zheng, L., & Cambria, E. (2021) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw53433.2021.00025 · Cited by 23
“[22] proposed a new framework, MISA, which projects each modality data into two different subspaces for multimodal fusion.”
BackgroundGe, S., Jiang, Z., Cheng, Z., Wang, C., Yin, Y., & Gu, Q. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583258 · Cited by 23
“Existing MMER methods generally focus on how to design efective network structures to obtain strong representation ability, and just naturally train the model on training data without intervention [1, 10, 13, 31].”
BackgroundA graph convolution-based heterogeneous fusion network for multimodal sentiment analysis
Zhao, T., Peng, J., Huang, Y., Wang, L., Zhang, H., & Cai, Z. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05151-w · Cited by 23
AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-Modal Alignment
Li, Y., Xing, Y., Lan, X., Li, X., Chen, H., & Jiang, D. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02307 · Cited by 30
“The resulting transport matrices provide interpretable alignment information between different modalities.”
Chen, C., Hong, H., Guo, J., & Song, B. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3263801 · Cited by 22
“MISA [4]employedmodality-invariantandspecificrepresen-tations.”
BackgroundLin, C., Cheng, H., Rao, Q., & Yang, Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3361374 · Cited by 24
“[8]projected eachmodalityintotwosubspaces.”
BackgroundInfluentialAssessing learners’ English public speaking anxiety with multimodal deep learning technologies
Zheng, C., Zhang, T., Chen, X., Zhang, H., Wan, J., & Wu, B. (2024) · Computer Assisted Language Learning · doi:10.1080/09588221.2024.2351129 · Cited by 28
“In order to verify the effectiveness and advancement of the model, we compared the MSAD with the existing evaluation model of PSA named Linguistic-Acoustic Decoupling (LAD) proposed by Song et al. (2023) and related affective computing models like Modality-Invariant and Modality-Specific Representations for Multimodal…”
InfluentialMulti-Grained Multimodal Interaction Network for Sentiment Analysis
Fang, L., Liu, G., & Zhang, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446351 · Cited by 21
“To evaluate the rationality and effectiveness of our methods, We compare proposed model with the following recent and competitive baselines: TFN [18], CIA [19], ICCN [20], PMR [5], MulT [4], MISA [7], MAG-BERT [21].”
BackgroundMethodResultInfluentialCorMulT: A Semi-Supervised Modality Correlation-Aware Multimodal Transformer for Sentiment Analysis
Li, Y., Zhu, R., & Li, W. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3559866 · Cited by 21
“However, most existing methods for multimoal sentiment analysis heavily rely on the assumption of strong correlations between modalities, and have not yet fully explored and examined the correlations between modalites.”
Wu, Y., Zhang, Z., Peng, P., Zhao, Y., & Qin, B. (2022) · doi:10.1145/3551876.3554813 · Cited by 21
“[16] first learn modality-invariant and modality-specific representations and then fuse them to predict the results.”
Exploring Semantic Relations for Social Media Sentiment Analysis
Zeng, J., Zhou, J., & Huang, C. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3285238 · Cited by 22
“For each sentence s i ∈ S , it contains up to k words: The sentiment score L of each review ranges from 1 to 5, and L ∈ [1 , 2 , 3 , 4 , 5] .”
BackgroundInfluentialZheng, Y., Gong, J., Wen, Y., & Zhang, P. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122728 · Cited by 23
Xue, J., Qin, R., Zhou, X., Liu, H., Zhang, M., & Zhang, Z. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446253 · Cited by 24
“Then, we compute orthogonal losses [22] among X audio , X text , and X fusion to optimize the fusion module.”
MethodHu, R., Yi, J., Chen, A., & Chen, L. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3388670 · Cited by 22
“…recurrent network (MARN) [31], memory fusion network (MFN) [32], recurrent attended variation embedding network (RAVEN) [33], MulT [19], MISA [15], the MAG-BERT [13], a multitask learning framework for Self-MM [16], an MAG+ [18], bidirection attention-based fusion network (BAFN) [34], CENet…”
BackgroundMethodInfluentialHiQuE: Hierarchical Question Embedding Network for Multimodal Depression Detection
Jung, J., Kang, C., Yoon, J., Kim, S., & Han, J. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679797 · Cited by 28
PAMoE-MSA: polarity-aware mixture of experts network for multimodal sentiment analysis
Huang, C., Lin, Z., Han, Z., Huang, Q., Jiang, F., & Huang, X. (2025) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-025-00362-y · Cited by 20
Toward Robust Multimodal Sentiment Analysis using multimodal foundational models
Zhao, X., Poria, S., Li, X., Chen, Y., & Tang, B. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126974 · Cited by 28
Self-citationDeep Multimodal Sequence Fusion by Regularized Expressive Representation Distillation
Guo, X., Kong, A. W., & Kot, A. C. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3142448 · Cited by 21
“Sub-space learning based methods such as MISA [23], explicitly projected multimodal representations onto modality-specific and modality-invariant subspaces.”
BackgroundMethodInfluentialStyleBERT: Text-audio sentiment analysis with Bi-directional Style Enhancement
Lin, F., Liu, S., Zhang, C., Jin, F., & Wu, Z. (2022) · Information Systems · doi:10.1016/j.is.2022.102147 · Cited by 20
Li, Y., Bell, P., & Lai, C. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832143 · Cited by 20
“• Modality-invariant and -specific fusion (MISA) [29]: combining both modality-invariant and modality-specific features.”
Dominant SIngle-Modal SUpplementary Fusion (SIMSUF) for Multimodal Sentiment Analysis
Huang, J., Ji, Y., Qin, Z., Yang, Y., & Shen, H. T. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3344358 · Cited by 21
“1(a) illustrates the traditional solutions for multimodal fusion, some frameworks balanced all modalities, using the same sub-network for all modality learning, and concatenated outputs for sentiment analysis [16], [51].”
BackgroundMethodInfluentialA dual-channel multimodal sentiment analysis framework based on three-way decision
Wang, X., Wang, M., Cui, H., & Zhang, Y. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109174 · Cited by 19
Injecting Multimodal Information Into Pre-Trained Language Model for Multimodal Sentiment Analysis
Mai, S., Zeng, Y., Xiong, A., & Hu, H. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3553149 · Cited by 21
“We compare our MGT with multiple competitive baselines, including Self-MM [66], TFN [8], LMF [71], MMIM [69], MISA [34], UniMSE [19], ConFEDE [49], ITHP [36], DMD [72], MM-DFN [73], MM-GCN [74], DAG-ERC [75], DialogueGCN [76], where UniMSE is the most similar to our approach.”
InfluentialEmoVerse: Enhancing Multimodal Large Language Models for Affective Computing via Multitask Learning
Li, A., Xu, L., Ling, C., Zhang, J., & Wang, P. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130810 · Cited by 26
Metadata
- Authors
- Hazarika, D., Zimmermann, R., & Poria, S.
- Affiliations
- National University of Singapore (SG)
- Singapore University of Technology and Design (SG)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 1122-1131
- OpenAlex
- W3093051361
- Cited by
- 1,073
- References
- 45
- Citations by year
- 2026: 280 · 2025: 315 · 2024: 238 · 2023: 136 · 2022: 78 · 2021: 23 · 2020: 3
- FWCI
- 22.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- Funders
- Ministry of Education - Singapore (MOE2018-T2-1-103)
- DSO National Laboratories - Singapore (RTDST190702)
- RIE 2020 Advanced Manufacturing and Engineering (AME) programmatic grant (A19E2b0098)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, modality gap, modality-invariant representations, modality-specific representations, subspace projection, multimodal fusion, user-generated video, humor detection, MOSI, MOSEI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
