The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Open accessAbstract
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On CMU-MOSEI dataset, our proposed C-MIB yields 0.7% improvement on Acc2 and 1.2% improvement on F1 score compared to the current state-of-the-art MISA [53]. The performance on Acc7 and Corr is also improved compared to the state-of-the-art methods.”
From Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations (2023) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 8×
“Inter-utterance context. et al. proposed one of the first models, bc-LSTM , which utilized”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“However, following recent works [7], including the stateof-the-art ICCN [50], we utilize the pre-trained BERT [11] as the feature extractor for textual utterances.”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“ndard low-level features that are provided by the respective benchmarks and utilized by the state-of-the-art methods. 4.3.1 Language Features. Traditionally, language modality features has been GloVe [38] embeddings for each token in the utterance. However, following recent works [7], including the stateof-the-art ICC…”
MethodInfluential(2008) · Journal of Machine Learning Research · Cited by 33,051
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 5×
“licly available models of [26, 51, 58]).”
BackgroundInfluentialSelf-citationEkman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,971 · cited 1×
“Both MOSI and MOSEI use Facet 3 to extract facial expression features, which include facial action units and face pose based on the Facial Action Coding System (FACS) [14].”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · cited 1×
“For UR_FUNNY, OpenFace [4], a facial behavioral analysis tool, is used to extract features related to the facial expressions of the speaker.”
Method
Research works printed but not linked: 15
- Andrew Galen, 2013, Proceedings of the 30th International Conference on Machine Learning, ICML 2013 (JMLR Workshop and Conference Proceedings, 1255
- Bousmalis Konstantinos, Advances in Neural Information Processing Systems 29. Curran Associates
- 10.5555/2354409.2355082 · DOI
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 19
- Drugman Thomas, 2011, Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonics. In INTERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association. ISCA
- Kiela Douwe, 2019, NeurIPS 2019 Workshop. Curran Associates, Inc.
- Lu Jiasen, Advances in Neural Information Processing Systems 32. Curran Associates
- 10.5555/2392963.2392965 · DOI
- Peng Minlong, 2018, Long Papers
- Rajagopalan Shyam Sundar, 2016, Proceedings, Part VII (Lecture Notes in Computer Science, 353
- Salzmann Mathieu, 2010, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010 (JMLR Proceedings, 708
- Hubert Tsai Yao-Hung, 2019, Learning Factorized Multimodal Representations. In 7th International Conference on Learning Representations, ICLR
- Vaswani Ashish, Advances in Neural Information Processing Systems 30. Curran Associates
- Zadeh Amir, 2018, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, 2246
- Zellinger Werner, 2017, Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning. In 5th International Conference on Learning Representations, ICLR
Cited by
Sorted by citations
Yu, W., Xu, H., Yuan, Z., & Wu, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17289 · Cited by 816
Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research
Poria, S., Hazarika, D., Majumder, N., & Mihalcea, R. F. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3038167 · Cited by 362
Self-citationHybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis
Mai, S., Zeng, Y., Zheng, S., & Hu, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3172360 · Cited by 307
UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition
Hu, G., Lin, T.-E., Zhao, Y., Lu, G., Wu, Y., & Li, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.534 · Cited by 304
Sun, L., Lian, Z., Liu, B., & Tao, J. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3274829 · Cited by 288
“Following previous works [16], [29], [32], we use the powerful BERT [52] model to encode raw text tokens into contextual word embeddings.”
MethodInfluentialBi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
Han, W., Chen, H., Gelbukh, A. F., Zadeh, A., Morency, L., & Poria, S. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479919 · Cited by 265
“To better illustrate how BBFN beats the SOTA (MISA), we compute the absolute errors of all the predictions on the test set of CMU-MOSEI and paint their distributions in Fig.”
BackgroundMethodInfluentialSelf-citationDecoupled Multimodal Distilling for Emotion Recognition
Li, Y., Wang, Y., & Cui, Z. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00641 · Cited by 249
“For a fair comparison with [7, 32] under the aligned setting, we additationally exploit a BERT-base-uncased pre-trained model [10] to obtain a 768-dimensional hidden state as the word features.”
InfluentialA Unified Multi-Task Semantic Communication System for Multimodal Data
Zhang, G., Hu, Q., Qin, Z., Cai, Y., Yu, G., & Tao, X. (2024) · IEEE Transactions on Communications · doi:10.1109/tcomm.2024.3364990 · Cited by 261
Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis
Yuan, Z., Li, W., Xu, H., & Yu, W. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475585 · Cited by 211
“MISA [10] learns modality-invariant and -specific representation for each modality to improve the fusion process.”
BackgroundMethodInfluentialExploiting BERT for Multimodal Target Sentiment Classification through Input Space Translation
Khan, Z., & Fu, Y. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475692 · Cited by 203
“Because multimodal sentiment analysis is a large and well-established field, we direct the reader to [2, 21, 29] for an overview of the field, and MISA [8], MAG [31], and M3ER [20] as representative of recent state of the art works.”
Cross-Modal Enhancement Network for Multimodal Sentiment Analysis
Wang, D., Liu, S., Wang, Q., Tian, Y., He, L., & Gao, X. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3183830 · Cited by 196
“Following prior works [2], [7], [8], we build two evaluation tasks including binary classification and regression.”
BackgroundMethodInfluentialMai, S., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3171679 · Cited by 182
“On CMU-MOSEI dataset, our proposed C-MIB yields 0.7% improvement on Acc2 and 1.2% improvement on F1 score compared to the current state-of-the-art MISA [53].”
MethodResultInfluentialState of the art: a review of sentiment analysis based on sequential transfer learning
Chan, J. Y.-L., Bea, K. T., Leow, S. M. H., Phoong, S. W., & Cheng, W. K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10183-8 · Cited by 197
“In addition, Hazarika et al. (2020) proposed a framework to better fuse the affective states to improve the modality representation.”
BackgroundCubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation
Sun, H., Wang, H., Liu, J., Chen, Y., & Lin, L. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548025 · Cited by 191
Latent Structure Mining With Contrastive Modality Fusion for Multimedia Recommendation
Zhang, J., Zhu, Y., Liu, Q., Zhang, M., Wu, S., & Wang, L. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3221949 · Cited by 189
“Previous work [33], [34], [35] notices that the modality-specific information could complement the modality-shared features captured in the invariant space and provides comprehensivemultimodal representations.”
BackgroundGCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation
Lian, Z., Chen, L., Sun, L., Liu, B., & Tao, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3234553 · Cited by 165
“We also utilize WAF as our evaluation metric, in line with previous works [59], [60].”
ResultTMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis
Huang, J., Zhou, J. W., Tang, Z., Lin, J., & Chen, C. Y. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111346 · Cited by 173
Li, J., Wang, X., Lv, G., & Zeng, Z. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3260635 · Cited by 146
“The experimental results on two benchmark datasets show that our GraphCFC outperforms the state-of-the-art (SOTA) approaches.”
BackgroundMultimodal Sentiment Detection Based on Multi-channel Graph Neural Networks
Yang, X., Feng, S., Zhang, Y., & Wang, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.28 · Cited by 145
Video sentiment analysis with bimodal information-augmented multi-head attention
Wu, T., Peng, J., Zhang, W., Zhang, H., Tan, S., Yi, F., Ma, C., & Huang, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107676 · Cited by 138
“Recently, multi-task learn-3 [55] 77.9 65.73 39.47 77.88 0.435 0.582 601K DFG[57] 78.77 65.65 39.82 78.21 0.445 0.578 2.7M MulT[41] 78.56 64.77 37.94 79.66 0.453 0.564 1.8M MISA[12] 69.”
BackgroundInfluentialRAFT: Robust Adversarial Fusion Transformer for multimodal sentiment analysis
Wang, R., Xu, D., Cascone, L., Wang, Y., Chen, H., Zheng, J., & Zhu, X. (2025) · Array · doi:10.1016/j.array.2025.100445 · Cited by 131
Multimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 118
Self-citationDynamic interactive multiview memory network for emotion recognition in conversation
Wen, J., Jiang, D., Tu, G., Liu, C., & Cambria, E. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.10.009 · Cited by 102
AF: An Association-Based Fusion Method for Multi-Modal Classification
Liang, X., Qian, Y., Guo, Q. J., Cheng, H., & Liang, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3125995 · Cited by 98
“Many recently proposed methods (e.g. [35], [69], [70], [71], and [64]) use the three datasets as benchmark datasets for evaluating their effectiveness.”
MethodMultimodal sentiment analysis: A survey
Lai, S., Hu, X., Xu, H., Ren, Z., & Liu, Z. (2023) · Displays · doi:10.1016/j.displa.2023.102563 · Cited by 106
Li, B., Fei, H., Liao, L., Zhao, Y., Teng, C., Chua, T., Ji, D., & Li, F. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612053 · Cited by 95
“While existing models, such as MISA [47] and FDMER [90], propose sophisticated disentanglement mechanisms for single pieces of utterance, disentangling on the conversational contexts has not been considered.”
MethodResultInfluentialLe, H.-D., Lee, G.-S., Kim, S.-H., Kim, S., & Yang, H.-J. (2023) · IEEE Access · doi:10.1109/access.2023.3244390 · Cited by 97
“Hazarika et al. [14] concentrated on enriching multimodal features before the fusion process by introducing MISA which learns modality-specific and modality-invariant representations for multimodal sentiment analysis.”
Multi-modal emotion recognition using tensor decomposition fusion and self-supervised multi-tasking
Wang, R., Zhu, J., Wang, S., Wang, T., Huang, J., & Zhu, X. (2024) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-024-00347-3 · Cited by 95
“Modality-Invariant and Modality-Specific Representations (MISA) [26] : MISA learns and fuses representations of different modalities by integrating distributional similarity, orthogonal loss, reconstruction loss, and task prediction loss.”
Liu, R., Zuo, H., Lian, Z., Schuller, B. W., & Li, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3378570 · Cited by 96
“In full-modality scenarios, some works tried to mine the intrinsic correlation of different modalities to alleviate the “ modality gap ” problem [16], [17], [18].”
InfluentialGkoumas, D., Li, Q., Lioma, C., Yu, Y., & Song, D. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.09.005 · Cited by 96
Chen, F., Shao, J., Zhu, S., & Shen, H. T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01036 · Cited by 98
Multimodal transformer with adaptive modality weighting for multimodal sentiment analysis
Wang, Y., He, J., Wang, D., Wang, Q., Wan, B., & Luo, X. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127181 · Cited by 104
SMIN: Semi-Supervised Multi-Modal Interaction Network for Conversational Emotion Recognition
Lian, Z., Liu, B., & Tao, J. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3141237 · Cited by 92
“Similar to MFM, MISA [54] factorizes each modality into two distinct subspaces.”
ResultBAFN: Bi-Direction Attention Based Fusion Network for Multimodal Sentiment Analysis
Tang, J., Liu, D., Jin, X., Peng, Y. K., Zhao, Q., Ding, Y., & Kong, W. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3218018 · Cited by 90
“Similarly, many attention-based methods, such as MISA [10] also used the softmax function to analyze the cross-modality interactions.”
BackgroundMethodInfluentialTeFNA: Text-centered fusion network with crossmodal attention for multimodal sentiment analysis
Huang, C., Zhang, J., Wu, X., Wang, Y., Li, M., & Huang, X. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110502 · Cited by 89
Wang, J. Z., Zhao, S., Wu, C., Adams, R. B., Newman, M. G., Shafir, T., & Tsachor, R. P. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3273517 · Cited by 109
Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities
Zeng, J., Liu, T., & Zhou, J. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3532064 · Cited by 94
“Previous works [14, 40] that directly utilized simple feed-forward neural layers with same parameters for the projection, which may be failed when there are more than two modalities.”
BackgroundMissModal: Increasing Robustness to Missing Modality in Multimodal Sentiment Analysis
Lin, R., & Hu, H. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00628 · Cited by 98
“The textual modality is viewed as the dominant modality in MSA task (Hazarika et al., 2020; Wu et al., 2021; Lin and Hu, 2023) due to the large-scale pre-trained language model and the nature of abundant semantic information instrumental in sentiment understanding.”
BackgroundCFN-ESA: A Cross-Modal Fusion Network With Emotion-Shift Awareness for Dialogue Emotion Recognition
Li, J., Wang, X., Liu, Y., & Zeng, Z. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3389453 · Cited by 83
“Moreover, there is a certain amount of noise in each modality itself, and together with the heterogeneity gap [20] of multimodal data, this direct concatenation manner may cause more noise.”
BackgroundAoM: Detecting Aspect-oriented Information for Multimodal Aspect-Based Sentiment Analysis
Zhou, R., Guo, W., Liu, X., Yu, S., Zhang, Y., & Yuan, X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.519 · Cited by 90
Brain-Machine Coupled Learning Method for Facial Emotion Recognition
Liu, D., Dai, W., Zhang, H., Jin, X., Cao, J., & Kong, W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3257846 · Cited by 83
“Similar to MISA [79], the preliminary representations are projected to two different channels, the common and private channels.”
BackgroundDisentanglement Translation Network for multimodal sentiment analysis
Zeng, Y., Yan, W., Mai, S., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102031 · Cited by 85
GPT-4V with emotion: A zero-shot benchmark for Generalized Emotion Recognition
Lian, Z., Sun, L., Sun, H., Chen, K., Wen, Z., Gu, H., Liu, B., & Tao, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102367 · Cited by 99
Contrastive Multimodal Fusion with TupleInfoNCE
Liu, Y., Fan, Q., Zhang, S., Dong, H., Funkhouser, T. A., & Yi, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00079 · Cited by 77
“Table 1 shows that the previous best performing method MISA [9] improves the segmentation mIoU by 3.”
BackgroundMethodInfluentialMultimodal Learning using Optimal Transport for Sarcasm and Humor Detection
Pramanick, S., Roy, A., & Johns, V. M. P. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00062 · Cited by 84
Sun, H., Chen, Y., & Lin, L. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3233070 · Cited by 81
“MISA [38] learns modality-invariant and modality-specific representations by projecting them into multiple subspaces.”
BackgroundZuo, H., Liu, R., Zhao, J., Gao, G., & Li, H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095836 · Cited by 75
Fan, C., Zhu, K., Tao, J., Yi, G., Xue, J., & Lv, Z. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3423671 · Cited by 81
“It is inappropriate for multimodal fusion to only consider reducing differences; the complementarity between modalities is equally important [57], [58], [59].”
Zeng, Y., Li, Z., Tang, Z., Chen, Z., & Ma, H. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119240 · Cited by 74
An effective multimodal representation and fusion method for multimodal intent recognition
Huang, X., Ma, T., Jia, L., Zhang, Y., Rong, H., & Al-Nabhan, N. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126373 · Cited by 72
Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimoda Emotion Recognition
Li, D., Wang, Y., Funakoshi, K., & Okumura, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.996 · Cited by 92
Multi-Task Momentum Distillation for Multimodal Sentiment Analysis
Lin, R., & Hu, H. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3282410 · Cited by 75
“MISA (Modality-Invariant and -Specific Representations) [3] projects features into modality-invariant and modality-specific MAG-BERT (Multimodal Adaption Gate for BERT) [11] dynamically shifts the verbal feature by nonverbal information to fine-tune the internal representations of the vanilla BERT…”
BackgroundMethodResultInfluentialCounterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis
Sun, T. F., Wang, W., Jing, L., Cui, Y., Song, X., & Nie, L. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548211 · Cited by 70
“Despite their compelling progress, existing studies usually suffer from fitting the spurious correlations in textual modality [8].”
BackgroundMethodInfluentialMIntRec: A New Dataset for Multimodal Intent Recognition
Zhang, H., Xu, H., Wang, X. Y., Zhou, Q., Zhao, S., & Teng, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547906 · Cited by 77
Tellamekala, M. K., Amiriparian, S., Schuller, B. W., André, E., Giesbrecht, T., & Valstar, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3325770 · Cited by 75
AtCAF: Attention-based causality-aware fusion network for multimodal sentiment analysis
Huang, C., Chen, J., Huang, Q., Wang, S., Tu, Y., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102725 · Cited by 76
Lu, Q., Sun, X., Gao, Z., Long, Y., Feng, J., & Zhang, H. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103538 · Cited by 69
Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning
Cai, Y., Li, X., Zhang, Y., Li, J., Zhu, F., & Rao, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-85859-6 · Cited by 68
Multimodal Phased Transformer for Sentiment Analysis
Cheng, J., Fostiropoulos, I., BOEHM, B. W., & Soleymani, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.189 · Cited by 69
“We perform a grid search for some of the hyper-parameters, consistent with previous work (Tsai et al., 2019; Rahman et al., 2020; Hazarika et al., 2020; Sun et al., 2020), and empirically select the remaining ones.”
BackgroundMethodResultInfluentialMultimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition
Guo, Z., Jin, T., & Zhao, Z. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.94 · Cited by 77
Cheng, H., Yang, Z., Zhang, X., & Yang, Y. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3265653 · Cited by 70
“(6) Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (MISA) [47].”
BackgroundLi, M., Yang, D., Zhao, X., Wang, S., Wang, Y., Yang, K., Sun, M., Kou, D., Qian, Z., & Zhang, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01184 · Cited by 78
“Mainstream MSA studies [9, 10, 22, 37, 50, 53, 55–58] focus on designing complex fusion paradigms and interaction mechanisms to enhance the performance of sentiment recognition.”
BackgroundDynamically Adjust Word Representations Using Unaligned Multimodal Information
Guo, J., Tang, J., Dai, W., Ding, Y., & Kong, W. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548137 · Cited by 65
Survey on multimodal approaches to emotion recognition
Aruna Gladys, A., & Vetriselvi, V. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126693 · Cited by 62
Contrastive-Based Removal of Negative Information in Multimodal Emotion Analysis
Wang, R., Wang, Y.-Y., Cambria, E., Fan, X.-H., Yu, X.-H., Huang, Y., Xiaosong, E., & Zhu, X.-X. (2025) · Cognitive Computation · doi:10.1007/s12559-025-10463-9 · Cited by 70
MEDT: Using Multimodal Encoding-Decoding Network as in Transformer for Multimodal Sentiment Analysis
Qi, Q., Lin, L., Zhang, R., & Xue, C. (2022) · IEEE Access · doi:10.1109/access.2022.3157712 · Cited by 66
“MISA : Modality-Invariant and -Specific Representations [20] project each modality to two subspaces of modal invariant and specific modalities to capture cross-modal commonality and unimodal private features for task prediction fusion.”
MethodResultInfluentialLearning to Learn Better Unimodal Representations via Adaptive Multimodal Meta-Learning
Ya, S., Mai, S., & Hu, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3178231 · Cited by 61
“Therefore, a recent work [29] presents a flexible multimodal learning framework, which utilizes invariant mappings to capture underlying common-”
BackgroundMethodInfluentialChen, Q., Huang, G., & Wang, Y. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3192728 · Cited by 58
“Hazarika et al. [12] proposed modality-specific representations and modality-invariant for multimodal representation learning.”
BackgroundMethodSemantic Alignment Network for Multi-Modal Emotion Recognition
Hou, M.-X., Zhang, Z., Liu, C., & Lu, G. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3247822 · Cited by 63
“ICCN [43], MISA [45], MSAF [46], Self-MM [44], and UEGD [47], as listed in Table IV, from which it can be concluded that our framework of multi-spatial semantic learning can also improve recognition in word-aligned data with the expansion of modalities.”
BackgroundTrustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space
Xie, Z., Yang, Y. Z., Wang, J., Liu, X., & Li, X. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3376564 · Cited by 65
“In addition, there are some multimodal fusion methods, such as graph-based fusion meth-ods [23], [24], [25], [26], tensor-based fusion methods [1], [27], and attention-based fusion methods [2], [28], [29].”
BackgroundMethodInfluentialExcavating multimodal correlation for representation learning
Mai, S., Ya, S., Zeng, Y., & Hu, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.003 · Cited by 57
Large language models meet text-centric multimodal sentiment analysis: a survey
Yang, H., Zhao, Y., Wu, Y., Wang, S., Zheng, T., Zhang, H., Ma, Z., Che, W., Wang, S., Wei, S., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4593-8 · Cited by 104
“Each query q has a corresponding key as k + while the other representations in the same batch are seen as k − .”
Lin, R., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3303711 · Cited by 59
“To further address the issues of exploding and vanishing gradients in parameters update, long short-term memory (LSTM) [20], [22] networks introduce input gates, forget gates, and output gates into RNNs, which is broadly used to learn visual and acoustic representations in MSA task [35], [51].”
BackgroundMethodResultMultimodal sentiment analysis with unimodal label generation and modality decomposition
Zhu, L., Zhao, H., Zhu, Z., Zhang, C., & Kong, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102787 · Cited by 55
Modality-invariant temporal representation learning for multimodal sentiment classification
Sun, H., Liu, J., Chen, Y., & Lin, L. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.10.031 · Cited by 57
TCHFN: Multimodal sentiment analysis based on Text-Centric Hierarchical Fusion Network
Hou, J., Omar, N., Tiun, S. B., Saad, S., & He, Q. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112220 · Cited by 55
Learning fine-grained representation with token-level alignment for multimodal sentiment analysis
Li, X., Zhang, H., Dong, Z., Cheng, X., Liu, Y., & Zhang, X. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126274 · Cited by 57
Huan, R., Zhong, G., Chen, P., & Liang, R. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3338769 · Cited by 60
“Additionally, we reproduce MISA [12], Self-MM [31], MMIM [32], MMIN [35], TFR-Net [23], and EMT-DLFR [39] using the same experimental setup for comparison.”
BackgroundMethodInfluentialA comprehensive survey on deep learning-based approaches for multimodal sentiment analysis
Ghorbanali, A., & Sohrabi, M. K. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10555-8 · Cited by 54
“This study used adversarial learning in a hybrid way to embed multimodal learning and does not have the complications of aligning the modalities (Hazarika et al. 2020; Mai et al. 2020 ).”
MethodShared-Private Memory Networks For Multimodal Sentiment Analysis
Zhao, X., Chen, Y., Liu, S., & Tang, B. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3222023 · Cited by 50
“Our work is inspired by the disentangled multimodal representation [9], [36] strategy.”
BackgroundMethodInfluentialUnderstanding and Constructing Latent Modality Structures in Multi-Modal Representation Learning
Jiang, Q., Chen, C., Zhao, H., Chen, L., Ping, Q., Tran, S. D., Xu, Y., Zeng, B., & Chilimbi, T. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00740 · Cited by 57
“Other works [20,58,69] investigate the modality-specific and modality-shared information.”
BackgroundDual-Perspective Fusion Network for Aspect-Based Multimodal Sentiment Analysis
Wang, D., Tian, C., Xiao, L., Zhao, L., He, L., & Wang, Q. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3321435 · Cited by 50
“For multimodal sentiment analysis tasks, recent approaches [22], [23], [24], [25], [26], [27], [28] use features from different modalities as transformer input for modality fusion.”
MethodModeling Hierarchical Uncertainty for Multimodal Emotion Recognition in Conversation
Chen, F., Shao, J., Zhu, A., Ouyang, D., Liu, X., & Shen, H. T. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3185119 · Cited by 50
“More recently, Hazarika et al. [47] explored enhancing cross-modal representation learning by projecting each modality to different subspaces.”
BackgroundLi, Z., Guo, Q., Pan, Y., Ding, W., Yu, J., Zhang, Y., Liu, W., Chen, H., Wang, H., & Xie, Y. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101891 · Cited by 49
Yang, D., Liu, Y., Huang, C., Li, M., Zhao, X., Wang, Y., Yang, K., Wang, Y., Zhai, P., & Zhang, L. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110370 · Cited by 47
MF-BERT: Multimodal Fusion in Pre-Trained BERT for Sentiment Analysis
He, J., & Hu, H. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3139856 · Cited by 48
MIA-Net: Multi-Modal Interactive Attention Network for Multi-Modal Affective Analysis
Li, S., Zhang, T., Chen, B., & Chen, C. L. P. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3259010 · Cited by 48
“Modality-invariant representation learning models contain MISA [48], MIB [47], MMIM [46].”
BackgroundMALN: Multimodal Adversarial Learning Network for Conversational Emotion Recognition
Ren, M.-J., Huang, X., Liu, J., Liu, M., Li, X., & Liu, A. (2023) · IEEE transactions on circuits and systems for video technology (Print) · doi:10.1109/tcsvt.2023.3273577 · Cited by 50
“Reference [23] projects each modality to two distinct subspaces to respectively learn the modality-invariant information and modality-specific information.”
BackgroundFMSA-SC: A Fine-Grained Multimodal Sentiment Analysis Dataset Based on Stock Comment Videos
Song, L., Chen, S., Meng, Z., Sun, M., & Shang, X. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3363641 · Cited by 49
Video-Based Cross-Modal Auxiliary Network for Multimodal Sentiment Analysis
Chen, R., Zhou, W., Li, Y., & Zhou, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3197420 · Cited by 46
“For example, both the MMIM [54] and MISA [55] apply specialized feature extrac-”
MethodRobust Multimodal Sentiment Analysis via Tag Encoding of Uncertain Missing Modalities
Zeng, J., Zhou, J., & Liu, T. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3207572 · Cited by 49
“Previous works [23], [24] utilized simple feed-forward neural layers with the same parameters to align different modalities, and then directly concatenated them to acquire multimodal representations.”
BackgroundMethodUnified Multi-modal Pre-training for Few-shot Sentiment Analysis with Prompt-based Learning
Yu, Y., Zhang, D., & Li, S. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548306 · Cited by 47
“, text and images as a tweet on social media [2, 7].”
BackgroundHybrid cross-modal interaction learning for multimodal sentiment analysis
Fu, Y., Zhang, Z., Yang, R., & Yao, C. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127201 · Cited by 46
A fine-grained modal label-based multi-stage network for multimodal sentiment analysis
Peng, J., Wu, T., Zhang, W., Cheng, F., Tan, S., Yi, F., & Huang, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119721 · Cited by 47
AIA-Net: Adaptive Interactive Attention Network for Text–Audio Emotion Recognition
Zhang, T., Li, S., Chen, B., Yuan, H., & Chen, C. L. P. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3195739 · Cited by 47
“Modality-Invariant Representation Learning Methods: Models that employ specific loss functions to learning common subspace during training contain: MISA [17], ICCN [41], and C-MIB [38].”
BackgroundMultimodal Mutual Attention-Based Sentiment Analysis Framework Adapted to Complicated Contexts
He, L., Wang, Z., Wang, L., & Li, F. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3276075 · Cited by 45
“On the basis of these studies, The authors in [31] and [32] have focused more on the specificity of multimodal sentiment analysis tasks or emotion recognition in conversation tasks.”
BackgroundWeakening the Dominant Role of Text: CMOSI Dataset and Multimodal Semantic Enhancement Network
Jin, C., Luo, C., Yan, M. G., Zhao, G., Zhang, G., & Zhang, S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3282953 · Cited by 44
“Hazarika et al. [11] mapped the features of different modes to different subspaces, which allowed the model to learn a more efficient modal representation and improve performance.”
Fu, Z., Liu, F., Xu, Q., Fu, X., & Qi, J. (2023) · Frontiers of Computer Science · doi:10.1007/s11704-023-2444-y · Cited by 44
Zhang, Q., Shi, L., Liu, P., Zhu, Z., & Xu, L. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03343-4 · Cited by 43
RetractedMSAF: Multimodal Split Attention FusionPreprint
Su, L., Hu, C., Li, G., & Cao, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07175 · Cited by 72
Metadata
- Authors
- Hazarika, D., Zimmermann, R., & Poria, S.
- Affiliations
- National University of Singapore (SG)
- Singapore University of Technology and Design (SG)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 1122-1131
- OpenAlex
- W3093051361
- Cited by
- 1,073
- References
- 45
- Citations by year
- 2026: 280 · 2025: 315 · 2024: 238 · 2023: 136 · 2022: 78 · 2021: 23 · 2020: 3
- FWCI
- 22.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- Funders
- Ministry of Education - Singapore (MOE2018-T2-1-103)
- DSO National Laboratories - Singapore (RTDST190702)
- RIE 2020 Advanced Manufacturing and Engineering (AME) programmatic grant (A19E2b0098)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, modality gap, modality-invariant representations, modality-specific representations, subspace projection, multimodal fusion, user-generated video, humor detection, MOSI, MOSEI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
