The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Open accessAbstract
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On CMU-MOSEI dataset, our proposed C-MIB yields 0.7% improvement on Acc2 and 1.2% improvement on F1 score compared to the current state-of-the-art MISA [53]. The performance on Acc7 and Corr is also improved compared to the state-of-the-art methods.”
From Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations (2023) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 8×
“Inter-utterance context. et al. proposed one of the first models, bc-LSTM , which utilized”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“However, following recent works [7], including the stateof-the-art ICCN [50], we utilize the pre-trained BERT [11] as the feature extractor for textual utterances.”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“ndard low-level features that are provided by the respective benchmarks and utilized by the state-of-the-art methods. 4.3.1 Language Features. Traditionally, language modality features has been GloVe [38] embeddings for each token in the utterance. However, following recent works [7], including the stateof-the-art ICC…”
MethodInfluential(2008) · Journal of Machine Learning Research · Cited by 33,051
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 5×
“licly available models of [26, 51, 58]).”
BackgroundInfluentialSelf-citationEkman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,971 · cited 1×
“Both MOSI and MOSEI use Facet 3 to extract facial expression features, which include facial action units and face pose based on the Facial Action Coding System (FACS) [14].”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · cited 1×
“For UR_FUNNY, OpenFace [4], a facial behavioral analysis tool, is used to extract features related to the facial expressions of the speaker.”
Method
Research works printed but not linked: 15
- Andrew Galen, 2013, Proceedings of the 30th International Conference on Machine Learning, ICML 2013 (JMLR Workshop and Conference Proceedings, 1255
- Bousmalis Konstantinos, Advances in Neural Information Processing Systems 29. Curran Associates
- 10.5555/2354409.2355082 · DOI
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 19
- Drugman Thomas, 2011, Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonics. In INTERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association. ISCA
- Kiela Douwe, 2019, NeurIPS 2019 Workshop. Curran Associates, Inc.
- Lu Jiasen, Advances in Neural Information Processing Systems 32. Curran Associates
- 10.5555/2392963.2392965 · DOI
- Peng Minlong, 2018, Long Papers
- Rajagopalan Shyam Sundar, 2016, Proceedings, Part VII (Lecture Notes in Computer Science, 353
- Salzmann Mathieu, 2010, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010 (JMLR Proceedings, 708
- Hubert Tsai Yao-Hung, 2019, Learning Factorized Multimodal Representations. In 7th International Conference on Learning Representations, ICLR
- Vaswani Ashish, Advances in Neural Information Processing Systems 30. Curran Associates
- Zadeh Amir, 2018, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, 2246
- Zellinger Werner, 2017, Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning. In 5th International Conference on Learning Representations, ICLR
Cited by
Sorted by citations
Items 201–300 of 1,001
M2H2: A Multimodal Multiparty Hindi Dataset For Humor Recognition in Conversations
Chauhan, D. S., Singh, G. V., Majumder, N., Zadeh, A., Ekbal, A., Bhattacharyya, P., Morency, L., & Poria, S. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479959 · Cited by 19
“For MISA+DialogueRNN, trimodal achieves the best precision of 71.21% ( 16 .”
MethodInfluentialSelf-citationMulti-View Interactive Representations for Multimodal Sentiment Analysis
Tang, Z., Xiao, Q., Qin, Y., Zhou, X., Zhou, J. T., & Li, K. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3357480 · Cited by 20
“By exploring these two types of information in modalities, effective multimodal representations can be learned [9], [21].”
BackgroundMethodInfluentialFMFN: A Fuzzy Multimodal Fusion Network for Emotion Recognition in Ensemble Conducting
Han, X., Chen, F., & Ban, J. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3373125 · Cited by 19
VAE-Based Adversarial Multimodal Domain Transfer for Video-Level Sentiment Analysis
Wang, Y., Wu, J., Furumai, K., Wada, S., & Kurihara, S. (2022) · IEEE Access · doi:10.1109/access.2022.3174215 · Cited by 19
Wang, H., Yang, M.-C., Li, Z., Liu, Z., Hu, J., Fu, Z.-W., & Liu, F. (2022) · Comput. Animat. Virtual Worlds · doi:10.1002/cav.2090 · Cited by 14
“MSA can be divided into two parts: unimodal representation 6,7 and multimodal fusion.”
BackgroundVersatile Multi-Modal Pre-Training for Human-Centric Perception
Hong, F., Pan, L., Cai, Z., & Liu, Z. (2022) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.01568 · Cited by 30
“Extensive studies [1, 15, 17, 34, 35, 41] focus on video-audio contrastive learning.”
BackgroundInter-Modality and Intra-Sample Alignment for Multi-Modal Emotion Recognition
Wang, Y., Li, D., & Shen, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446571 · Cited by 20
“By doing so, we can extract valuable potential characteristics from each modality that are synergistic and contribute to overall affective orientation [12].”
BackgroundAn Image-Text Sentiment Analysis Method Using Multi-Channel Multi-Modal Joint Learning
Gong, L., He, X., & Jianzhong, Y. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2371712 · Cited by 17
SDRS: Sentiment-Aware Disentangled Representation Shifting for Multimodal Sentiment Analysis
Zhao, S., Yang, Z., Shi, H., Feng, X., Meng, L., Qin, B., Yan, C. C., Tao, J., & Ding, G. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3539225 · Cited by 18
“Hazarika et al. [41] decomposed multimodal features into modality-invariant/specific components to learn the refined multimodal representations.”
Cross-modality Representation Interactive Learning for Multimodal Sentiment Analysis
Huang, J., Ji, Y., Yang, Y., & Shen, H. T. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612295 · Cited by 17
Modality-collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition
Chen, C., & Zhang, P. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3640343 · Cited by 18
“By projecting each sample’s modality into two subspaces, this method learns both Modality-Invariant and -Specific Representations (MISA) [27] which benefit the multimodal fusion.”
BackgroundMethodInfluentialCross-Modal Dynamic Transfer Learning for Multimodal Emotion Recognition
Hong, S., Kang, H., & Cho, H. (2024) · IEEE Access · doi:10.1109/access.2024.3356185 · Cited by 20
“MISA [11] is one of the multimodal fusion models that focuses on the aspect of the modality representation space for sentiment analysis tasks.”
BackgroundMethodInfluentialMeta Noise Adaption Framework for Multimodal Sentiment Analysis With Feature Noise
Yuan, Z., Zhang, B., Xu, H., & Gao, K. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3362600 · Cited by 19
“In particular, typical methods including the Multimodal Transformer (MulT) [53], the Modality-Invariant and -Specific Representations (MISA) [54], and the Self-supervised Multi-task Multimodal sentiment analysis network (Self-MM) [39] are first selected.”
MethodYin, B., Chen, Z., & Tao, M. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3364673 · Cited by 17
“For example, human action recognition [14] and multi-modal sentiment analysis [15] can extract multi-modal feature representation from different modalities including audio and vision, thereby improving prediction performance.”
BackgroundMERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition
Lian, Z., Sun, L., Ren, Y., Gu, H., Sun, H., Chen, L., Liu, B., & Tao, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3653457 · Cited by 71
“The above experiments primarily utilize pre-trained models as feature extractors and then use shallow classifiers for emotion recognition.”
In Your Eyes: Modality Disentangling for Personality Analysis in Short Video
Sun, X., Liu, B., Ai, L., Liu, D., Meng, Q., & Cao, J. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3161708 · Cited by 17
“7) MISA [46]: This method is aimed at video emotion analysis.”
MethodSense-aware BERT and Multi-task Fine-tuning for Multimodal Sentiment Analysis
Fang, L., Liu, G., & Zhang, R. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892116 · Cited by 16
“Some previous work has only used BERT for word representation in multimodal learning [3], which is far from exploiting the performance of BERT.”
BackgroundMethodInfluentialAttA-NET: Attention Aggregation Network for Audio-Visual Emotion Recognition
Fan, R., Liu, H., Li, Y., Guo, P., Wang, G., & Wang, T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447640 · Cited by 16
“In multi-modal tasks, features exhibiting modal-shared and modal-specific properties are expected [7].”
BackgroundLai, S., Li, J., Guo, G., Hu, X., Li, Y., Tan, Y., Song, Z., Liu, Y., Ren, Z., Wang, C., et al. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651442 · Cited by 16
Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems
Lin, T.-E., Wu, Y., Huang, F., Si, L., Sun, J., & Li, Y. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539209 · Cited by 17
Zhao, X., Li, X., Jiang, R., & Tang, B. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102745 · Cited by 18
Zhu, M., He, X., Qiao, B., Luo, Y., Li, Z., & Pan, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113249 · Cited by 15
Joint Modality Features in Frequency Domain for Stress Detection
Radhika, K. R., Subramanian, R., & Murthy, O. V. R. (2022) · IEEE Access · doi:10.1109/access.2022.3178409 · Cited by 15
“A modality-invariant (MI) representations for multimodal sentiment analysis was proposed by Hazarika et al. [14].”
MethodPace-Adaptive and Noise-Resistant Contrastive Learning for Multimodal Feature Fusion
Guo, X., Kot, A. C., & Kong, A. W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3252270 · Cited by 17
“At the same time, from the perspective of multimodal fusion [17], [21], [25], [26], [27], [28], [29], modality-invariant information [30] is expected to be maintained and maximized, but this problem is largely unexplored in MCL.”
BackgroundTMMDA: A New Token Mixup Multimodal Data Augmentation for Multimodal Sentiment Analysis
Zhao, X., Chen, Y., Liu, S., Zang, X., Xiang, Y., & Tang, B. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583406 · Cited by 15
OSAN: A One-Stage Alignment Network to Unify Multimodal Alignment and Unsupervised Domain Adaptation
Liu, Y., Qiao, L., Lu, C., Yin, D., Chen, L., Peng, H., & Ren, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00346 · Cited by 15
Multi-Modal Emotion Recognition for Enhanced Requirements Engineering: A Novel Approach
Cheng, B., Arora, C., Liu, X., Hoang, T. N., Wang, Y., & Grundy, J. (2023) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re57278.2023.00039 · Cited by 16
“Recent advances in Artificial Intelligence (AI) and machine learning have led to significant improvements in emotion recognition techniques in videos, audios, and texts [6]–[8].”
BackgroundDeep Modular Co-Attention Shifting Network for Multimodal Sentiment Analysis
Shi, P., Hu, M. Q., Shi, X., & Ren, F. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3634706 · Cited by 15
“Hazarika et al. used modality-invariant and modality-specific features to reduce the modality gap and predict emotion states [15].”
BackgroundKEBR: Knowledge Enhanced Self-Supervised Balanced Representation for Multimodal Sentiment Analysis
Zhu, A., Hu, M. Q., Wang, X., Yang, J., Tang, Y., & Ren, F. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681163 · Cited by 18
“Therefore, in the testing phase, we only added an output layer to the pre-trained language model as well as the multimodal fusion module to generate task-specific predictions.”
InfluentialUtilizing BERT Intermediate Layers for Multimodal Sentiment Analysis
Zou, W., Ding, J., & Wang, C. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9860014 · Cited by 17
“1): (a) Hazarika et al. use BERT to extract text features, and then use their designed network [7] to process multimodal features; (b) Rahman et al. integrate non-verbal information on the one of layers of BERT by a middleware called Multi-modal Adaptive Gate (MAG) [8] and implement downstream…”
MethodInfluentialZhao, X., Chen, Y., Chen, Y.-T., Liu, S., & Tang, B. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859747 · Cited by 14
“…[18], Multimodal Adaptation Gate BERT (MAG-BERT) [4], Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (MISA) [9], Modal-Temporal Attention Graph (MTAG) [17], Progressive Modality Reinforcement (PMR) [7] and Self-Supervised Multi-task Multimodal sentiment…”
MethodCLAP: Contrastive Language-Audio Pre-training Model for Multi-modal Sentiment Analysis
Zhao, T., Kong, M. C., Liang, T., Zhu, Q., Kuang, K., & Wu, F. Q. (2023) · International Conference on Multimedia Retrieval · doi:10.1145/3591106.3592296 · Cited by 16
“To evaluate the effectiveness of our proposed CLAP modal, we compare our sentiment prediction performance with the classic MSA baselines such as TFN [42], LFM [17], MFM [34], ICCN [31], MulT [33] and MISA [8].”
MethodZero-shot multitask intent and emotion prediction from multimodal data: A benchmark study
Singh, G. V., Firdaus, M., Chauhan, D. S., Ekbal, A., & Bhattacharyya, P. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127128 · Cited by 15
Li, J., Cai, S., Li, L., Sun, R., Yuan, G., & Zhu, R. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123655 · Cited by 14
SIA-Net: Sparse Interactive Attention Network for Multimodal Emotion Recognition
Li, S., Zhang, T., & Chen, C. L. P. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3409715 · Cited by 14
“MISA [29] factorized multimodal features into modality-invariant and modality-specific features through central moment discrepancy loss and difference loss, respectively.”
BackgroundMual: enhancing multimodal sentiment analysis with cross-modal attention and difference loss
Deng, Y., Li, Y., Xian, S., Li, L., & Qiu, H. (2024) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-024-00340-w · Cited by 14
“Following the approach of [21], the weight coefficient α is set to 0.3.”
InfluentialMultifaceted and deep semantic alignment network for multimodal sarcasm detection
Yu, B., Wang, H., & Xi, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112298 · Cited by 16
A unified multimodal classification framework based on deep metric learning
Peng, L., Jian, S., Li, M., Kan, Z., Qiao, L., & Li, D. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106747 · Cited by 16
Fu, J., Fu, Y., Xue, H., & Xu, Z. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01724-5 · Cited by 16
“• MISA [17]projectsfeaturesintodistincttwo-dimensional spaces that are subject to specific constraints and then conducts fusion on these features.”
AFR-BERT: Attention-based mechanism feature relevance fusion multimodal sentiment analysis model
Mingyu, J., Zhou, J., & Ning, W. (2022) · PLoS ONE · doi:10.1371/journal.pone.0273936 · Cited by 16
TriagedMSA: Triaging Sentimental Disagreement in Multimodal Sentiment Analysis
Luo, Y., Liu, W., Sun, Q., Li, S., Li, J., Wu, R., & Tang, X. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3524789 · Cited by 15
“Modality invariant and specific representations (MISA) [18]: This method uses the loss function to learn the common and characteristic features between unimodal representations, and uses the transformer to fuse them.”
InfluentialMultimodal hypergraph network with contrastive learning for sentiment analysis
Huang, J., Jiang, K., Pu, Y., Zhao, Z., Yang, Q., Gu, J., & Xu, D. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129566 · Cited by 15
Hierarchical multi-source cues fusion for mono-to-binaural based Audio Deepfake Detection
Liu, R., Zhang, J., & Li, H. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103097 · Cited by 17
Adaptive multimodal transformer based on exchanging for multimodal sentiment analysis
Tuerhong, G., Fu, F., & Wushouer, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-11848-4 · Cited by 13
Multi-Level Contrastive Learning for Multimodal Sentiment Analysis
Yan, Z., Bai, W., Zhang, Y., Deng, J., Hu, Z., Zhang, X., & Ren, F. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3613116 · Cited by 16
“These models include: TFN [26], LMF [27], MFN [80], MulT [4], MISA [5], ICCN [81], MAG-BERT [82], SELF-MM [83], MSG [62], LNLN [84], GCNet [85], MPLMM [86], DiC-MoR [66], IMDer [77], AGM [87], MIL [88], and BCFN [89].”
InfluentialScanning, attention, and reasoning multimodal content for sentiment analysis
Liu, Y., Li, Z., Zhou, K., Zhang, L., Li, L., Tian, P., & Shen, S. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110467 · Cited by 13
Liu, P., Zheng, X. W., Li, H., Liu, J., Ren, Y., Zhu, H., & Sun, L. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096470 · Cited by 13
“MulT [6] proposes the multimodal transformer, where a cross-modal attention module is learned for reinforcing one MISA [16] factorizes each modal information into modality-invariant and modality-specific to learn common features across modalities and reduce the modality gap.”
BackgroundMethodLearning Cross-Modal Audiovisual Representations with Ladder Networks for Emotion Recognition
Goncalves, L., & Busso, C. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096138 · Cited by 13
“In backward methods, studies utilize extra loss functions as a prior constraint to obtain complementary information between the modalities [8, 12].”
BackgroundMethodLow-rank tensor fusion and self-supervised multi-task multimodal sentiment analysis
Miao, X., Zhang, X., & Zhang, H. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-18032-8 · Cited by 13
“For audio and video, as in previous work by Hazarika et al. [38] and Yu et al. [21], we used the pre-trained toolkit CMU-Multimodal-SDK to extract the initial vector features from the raw data X a and X v , but this paper differs from previous work in that instead of utilizing unidirectional LSTM…”
Lu, N., Tan, Z., & Qian, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127467 · Cited by 13
Yang, D., Li, M., Qu, L., Yang, K., Zhai, P., Wang, S., & Zhang, L. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3435561 · Cited by 16
“Despite recent impressive advancements, challenges remain due to unpreventable temporal asynchrony and modality heterogeneity.”
Graph Reconstruction Attention Fusion Network for Multimodal Sentiment Analysis
Hu, R., Yi, J., Chen, L., & Jin, Z. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3452204 · Cited by 13
“For sequence learning, MISA [5] projects modals into different subspaces to learn specific/invariant represen-tationsofthem.”
WavFusion: Towards Wav2vec 2.0 Multimodal Speech Emotion Recognition
Li, F., Luo, J., & Xia, W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-2071-5_24 · Cited by 13
Domain-Separated Bottleneck Attention Fusion Framework for Multimodal Emotion Recognition
He, P., Yu, J., Ge, C., Yu, Y., Xu, W., Wang, L., Liu, T., & Kan, Z. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3711865 · Cited by 16
Learning Explicit and Implicit Dual Common Subspaces for Audio-visual Cross-modal Retrieval
Zeng, D.-H., Wu, J., Hattori, G., Xu, R., & Yu, Y. (2022) · ACM Trans. Multim. Comput. Commun. Appl. · doi:10.1145/3564608 · Cited by 17
“Since the video feature extraction is extremely useful for several video-relevant applications, several studies have applied audio-visual joint embedding learning to a number of tasks such as multimodal sentiment analysis [21], audio-visual separation [14, 32], audio-visual location [60, 73, 74], visual and sound reco…”
BackgroundMulti-level fusion with fine-grained alignment for multimodal sentiment analysis
Li, X., Ma, Y., An, X., Xing, J., Liu, R. P., & Ren, Y. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00094-3 · Cited by 12
“…they often incorporate regularization strategies like self-supervised learning, mutual information maximization (Han et al. 2021), multi-task (Hazarika et al. 2020; Tang et al. 2022; Yu et al. 2023; Cai et al. 2025) learning, contrast learning (Mai et al. 2023), feature decomposition (Wang…”
Visual Sentiment Analysis Using Machine Learning for Entertainment Applications
Sayal, A., Chaithra, N., Jha, J., Trilochan, B., Kalyan, G., Priya, M., Gupta, V., Memoria, M., & Gupta, A. (2023) · International Conference on IoT, Communication and Automation Technology (ICICAT) · doi:10.1109/icicat57735.2023.10263629 · Cited by 12
“The MISA framework [34] attempts to enhance multimodal sentiment analysis by using factorized subspaces learned for each modality.”
BackgroundBrain-Inspired Recognition System Based on Multimodal In-Memory Computing Framework for Edge AI
Xiao, H., Xie, D., Hu, X., Zhou, Y., Zhou, G., & Duan, S. (2023) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2023.3337532 · Cited by 12
“To evaluate the effectiveness of our proposed system in MSA and ERC tasks, we compared it with 16 baseline meth-ods [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62].”
MethodBalanced sentimental information via multimodal interaction model
Luo, Y., Wu, R., Liu, J., & Tang, X. (2024) · Multimedia Systems · doi:10.1007/s00530-023-01208-5 · Cited by 12
Multi-Modal Hierarchical Empathetic Framework for Social Robots With Affective Body Control
Gao, Y., Fu, Y., Sun, M., & Gao, F. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3356511 · Cited by 12
A Two-Stage Stacked Transformer Framework for Multimodal Sentiment Analysis
Yi, G., Fan, C., Tao, J., Lv, Z., Wen, Z., Pei, G., & Li, T. (2024) · Intelligent Computing · doi:10.34133/icomputing.0081 · Cited by 12
CCMA: CapsNet for audio–video sentiment analysis using cross-modal attention
Li, H., Guo, A., & Li, Y. (2024) · The Visual Computer · doi:10.1007/s00371-024-03453-9 · Cited by 12
“MISA [33] projected each modality separately into the shared and private space to potentially learn modal commonalities and narrow the modality gap.”
Liu, W., Xu, H., Hua, Y., Chi, Y., & Gao, K. (2024) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2024.200410 · Cited by 13
Li, Z., Liu, P., Pan, Y., Yu, J., Liu, W., Chen, H., Luo, Y., & Wang, H. (2024) · Applied Intelligence · doi:10.1007/s10489-024-06150-1 · Cited by 14
Dong, Z., Hu, C., Zhu, L., Ji, X., & Lai, C. S. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3527206 · Cited by 13
“The cross-modal attention (CMA) fusion [34], MISA fusion [35], transformer-based fusion [36], and learned weights fusion [37] are applied to replace the concatenation unit, the obvious reduction can be witnessed in classification performance (CMA fusion: − 3.55% Acc-7, − 3.85% F1-7, − 4.59% Acc-3, −…”
InfluentialMulti-level feature decomposition and fusion model for video-based multimodal emotion recognition
Liu, X., He, G.-T., Li, S., Yang, F., He, S., & Chen, L. (2025) · Engineering applications of artificial intelligence · doi:10.1016/j.engappai.2025.110744 · Cited by 13
TEMSA:Text enhanced modal representation learning for multimodal sentiment analysis
Chen, J., Song, S., Tan, Y., & Xia, H. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104391 · Cited by 13
Uncertain Multimodal Intention and Emotion Understanding in the Wild
Yang, Q., Shi, Q., Wang, T., & Ye, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02300 · Cited by 13
The ReprGesture entry to the GENEA Challenge 2022
Yang, S., Wu, Z., Li, M., Zhao, M., Lin, J., Chen, L.-Y., & Bao, W. (2022) · International Conference on Multimodal Interaction · doi:10.1145/3536221.3558066 · Cited by 13
“Inspired by [19] and [6], we propose a gesture generation system based on multimodal representation learning.”
MethodMultimodal interaction enhanced representation learning for video emotion recognition
Xia, X., Zhao, Y. H., & Jiang, D. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.1086380 · Cited by 12
Adapted Multimodal Bert with Layer-Wise Fusion for Sentiment Analysis
Chlapanis, O. S., Paraskevopoulos, G., & Potamianos, A. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094923 · Cited by 13
VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning
Wang, Y., Zeng, D., Wada, S., & Kurihara, S. (2023) · IEEE Access · doi:10.1109/access.2023.3280187 · Cited by 13
“high-performance text sentiment analysis [2].”
BackgroundMethodEmotion Dictionary Learning With Modality Attentions for Mixed Emotion Exploration
Liu, F., Yang, P., Shu, Y., Yan, F., Zhang, G., & Liu, Y. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3334520 · Cited by 11
Nguyen, C.-D. T., Nguyen, T. T., Vu, D., & Luu, A. N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.980 · Cited by 15
Hierarchical Multi-Modal Prompting Transformer for Multi-Modal Long Document Classification
Liu, T., Hu, Y., Gao, J., Sun, Y., & Yin, B. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3366935 · Cited by 11
“For the multi-modal baselines, we consider three categories of methods: multi-modal classification [53], [54], [55], [58], multi-modal emotion analysis [56], [57], and multi-modal transformer [9], [15].”
MethodA Fine-Grained Tri-Modal Interaction Model for Multimodal Sentiment Analysis
Zhi, Y., Li, J., Wang, H., Chen, J., & Cao, T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447872 · Cited by 11
“Inspired by MISA[9], we designed a fully connected layer with learnable parameters θ G , mapping each embeddings to the same potential subspace to pull in the distance of different modalities.”
BackgroundMethodInfluentialMultimodal Consistency-Based Teacher for Semi-Supervised Multimodal Sentiment Analysis
Yuan, Z., Fang, J., Xu, H., & Gao, K. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3430543 · Cited by 14
“The Modality-Invariant and -Specific Representations [35] is made up of a combination of losses including similarity loss, orthogonal loss, reconstruction loss and prediction loss to learn modality-invariant and modality-specific representation.”
BackgroundPrompt Learning for Multimodal Intent Recognition with Modal Alignment Perception
Chen, Y., Zhu, W., Yu, W., Xue, H., Fu, H., Lin, J., & Jiang, D. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10328-7 · Cited by 12
“MISA (Hazarika et al. [10]) obtains modality-invariant and modality-specific features from each modality and then fuses them with the self-attention mechanism.”
InfluentialRobust Multimodal Representation under Uncertain Missing Modalities
Lan, G., Du, Y., & Yang, Z. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3702003 · Cited by 11
Zhu, Z., Cheng, X., Chen, Z., Chen, Y., Zhang, Y., Wu, X., Zheng, Y., & Xing, B. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681623 · Cited by 13
Reward-Based Gradient Modulation for Multimodal Emotion Recognition With LoRA
Liu, F., Fu, Z., & Wang, Y. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3566373 · Cited by 11
“MISA [40].”
Wei, T., Chen, H., Liu, W., Chen, L., & Wang, J. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3591926 · Cited by 15
“…i )di ff are obtained as follows: Next, the modality-share features are obtained by mapping multimodal features into a common embedding space [49], [50] using shared-weight VSS blocks, which scan and merge multimodal features pixel by pixel to capture shared land cover information perceivable by…”
Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos
Ma, L., Yu, Y., Liang, T., & Liu, T. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0351-0_21 · Cited by 11
On the Use of Modality-Specific Large-Scale Pre-Trained Encoders for Multimodal Sentiment Analysis
Ando, A., Masumura, R., Takashima, A., Suzuki, S., Makishima, N., Suzuki, K., Moriya, T., Ashihara, T., & Satō, H. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022548 · Cited by 12
“Results of †, ‡ and ⋄ are from [12], [10], and [13], respectively.”
BackgroundMethodTransformer-based adaptive contrastive learning for multimodal sentiment analysis
Hu, Y., Huang, X., Wang, X., Lin, H., & Zhang, R. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19152-5 · Cited by 13
Multi-modal long document classification based on Hierarchical Prompt and Multi-modal Transformer
Liu, T., Hu, Y., Gao, J., Wang, J., Sun, Y., & Yin, B. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106322 · Cited by 11
Lv, Z., Yan, R., Lin, Y., Gao, L., Zhang, F., & Wang, Y. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020012 · Cited by 10
Exploring Multimodal Sentiment Analysis Models: A Comprehensive Survey
Quoc, P. D., Nguyen-Tat, T. B., Roantree, M., & Ngo, V. M. (2024) · International Conference on Multimedia Analysis and Pattern Recognition (MAPR) · doi:10.1109/mapr63514.2024.10660747 · Cited by 10
“The new proposed Hyperbolic Hierarchical Attention Network model showed semantic similarity between written and visual content when compared to the summary.”
Jing, Y., & Zhao, X. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681599 · Cited by 10
Adaptive Multimodal Graph Integration Network for Multimodal Sentiment Analysis
Gong, P., Liu, J., Zhang, X., Li, X., Wei, L., & He, H. (2024) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3507576 · Cited by 11
“Included in this group are models like RAVEN [20], MCTN [49], Self-MM [25], MISA [22], C-MIB [50] and HyCon [51].”
Kim, T., Yang, J., & Park, E. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521707 · Cited by 11
“To quantify the similarities and differences between text and speech features, we utilize a loss function, which enables the representation of these modalities [52].”
An Enhanced Cross‐Attention Based Multimodal Model for Depression Detection
Kou, Y., Ge, F., Chen, D., Shen, L., & Liu, H. (2025) · Computational Intelligence · doi:10.1111/coin.70019 · Cited by 10
RSNet: Region-Specific Network for Contactless Palm Vein Authentication
Luo, D., Huang, J., Yang, W., Shakeel, M. S., & Kang, W. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3544029 · Cited by 10
“The di ff erence loss [32] has been successfully used in [52] and [53] to learn di ff erent aspects of the features.”
Wen, Y., Li, Q., Zhou, Y., Gao, Y., Wen, Z., Tao, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3578618 · Cited by 12
“To comprehensively validate the performance of our model, we compare our method with several advanced and SOTA baselines in Tables II and III: TFN [30], LMF [31], MulT [14], MAG-BERT [57], MISA [58], Self-MM [48], MMIM [32], and ConFEDE [43].”
Speech-Visual Emotion Recognition by Fusing Shared and Specific Features
Chen, G., & Jiao, S. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3279689 · Cited by 10
“[26] only applies the similarity loss function to obtain the shared features of speech and visual modalities, due to the heterogeneity between speech and visual modalities, which is difficult to obtain high quality shared features for speech and visual modalities, and thus affects the performance of [26].”
BackgroundInvariant Meets Specific: A Scalable Harmful Memes Detection Framework
Yang, C., Zhu, F., Han, J., & Hu, S. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611761 · Cited by 10
Joint Objective and Subjective Fuzziness Denoising for Multimodal Sentiment Analysis
Jiang, X., Xu, X., Lu, H., He, L., & Shen, H. T. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3405541 · Cited by 10
Xu, H. (2023) · Springer eBooks · doi:10.1007/978-981-99-5776-7_6 · Cited by 9
Multimodal Sentiment Analysis Method Based on Hierarchical Adaptive Feature Fusion Network
Zhang, H. (2024) · International Journal on Semantic Web and Information Systems · doi:10.4018/ijswis.335918 · Cited by 9
“The following is a comparative of the proposed model with TFN (Zadeh et al., 2017), MulT (Tsai et al., 2019), MISA (Hazarika et al., 2020), Self-MM (Yu et al., 2021), and ConFEDE (Yang et al., 2023) based on the CH-SIMS dataset: (1) TFN (Zadeh et al., 2017): First, create multidimensional tensors…”
BackgroundMethodInfluentialYu, B., & Shi, Z. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06422-0 · Cited by 10
“Hazarika et al. [26] provided a holistic view of multimodal data by projecting individual modalities into the modal invariant and modal specific sub-spaces.”
Metadata
- Authors
- Hazarika, D., Zimmermann, R., & Poria, S.
- Affiliations
- National University of Singapore (SG)
- Singapore University of Technology and Design (SG)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 1122-1131
- OpenAlex
- W3093051361
- Cited by
- 1,073
- References
- 45
- Citations by year
- 2026: 280 · 2025: 315 · 2024: 238 · 2023: 136 · 2022: 78 · 2021: 23 · 2020: 3
- FWCI
- 22.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- Funders
- Ministry of Education - Singapore (MOE2018-T2-1-103)
- DSO National Laboratories - Singapore (RTDST190702)
- RIE 2020 Advanced Manufacturing and Engineering (AME) programmatic grant (A19E2b0098)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, modality gap, modality-invariant representations, modality-specific representations, subspace projection, multimodal fusion, user-generated video, humor detection, MOSI, MOSEI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
