The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Open accessAbstract
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On CMU-MOSEI dataset, our proposed C-MIB yields 0.7% improvement on Acc2 and 1.2% improvement on F1 score compared to the current state-of-the-art MISA [53]. The performance on Acc7 and Corr is also improved compared to the state-of-the-art methods.”
From Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations (2023) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 8×
“Inter-utterance context. et al. proposed one of the first models, bc-LSTM , which utilized”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“However, following recent works [7], including the stateof-the-art ICCN [50], we utilize the pre-trained BERT [11] as the feature extractor for textual utterances.”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“ndard low-level features that are provided by the respective benchmarks and utilized by the state-of-the-art methods. 4.3.1 Language Features. Traditionally, language modality features has been GloVe [38] embeddings for each token in the utterance. However, following recent works [7], including the stateof-the-art ICC…”
MethodInfluential(2008) · Journal of Machine Learning Research · Cited by 33,051
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 5×
“licly available models of [26, 51, 58]).”
BackgroundInfluentialSelf-citationEkman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,971 · cited 1×
“Both MOSI and MOSEI use Facet 3 to extract facial expression features, which include facial action units and face pose based on the Facial Action Coding System (FACS) [14].”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · cited 1×
“For UR_FUNNY, OpenFace [4], a facial behavioral analysis tool, is used to extract features related to the facial expressions of the speaker.”
Method
Research works printed but not linked: 15
- Andrew Galen, 2013, Proceedings of the 30th International Conference on Machine Learning, ICML 2013 (JMLR Workshop and Conference Proceedings, 1255
- Bousmalis Konstantinos, Advances in Neural Information Processing Systems 29. Curran Associates
- 10.5555/2354409.2355082 · DOI
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 19
- Drugman Thomas, 2011, Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonics. In INTERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association. ISCA
- Kiela Douwe, 2019, NeurIPS 2019 Workshop. Curran Associates, Inc.
- Lu Jiasen, Advances in Neural Information Processing Systems 32. Curran Associates
- 10.5555/2392963.2392965 · DOI
- Peng Minlong, 2018, Long Papers
- Rajagopalan Shyam Sundar, 2016, Proceedings, Part VII (Lecture Notes in Computer Science, 353
- Salzmann Mathieu, 2010, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010 (JMLR Proceedings, 708
- Hubert Tsai Yao-Hung, 2019, Learning Factorized Multimodal Representations. In 7th International Conference on Learning Representations, ICLR
- Vaswani Ashish, Advances in Neural Information Processing Systems 30. Curran Associates
- Zadeh Amir, 2018, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, 2246
- Zellinger Werner, 2017, Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning. In 5th International Conference on Learning Representations, ICLR
Cited by
Sorted by citations
Items 601–700 of 1,001
Multimodal Sentiment Analysis: A Co-occurrence Network-Based Bibliometric Analysis
Kulkarni, A. S., Acharya, S., Paul, P., & Misra, B. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-95-1069-6_5 · Cited by 1
Balamurugan, V., & Sivasankari, K. (2025) · International Conference on New Frontiers in Communication, Automation, Management and Security (ICCAMS) · doi:10.1109/iccams65118.2025.11234648 · Cited by 1
“In this work, convolution neural network with skip connection referred as U-Net[10]is employed as pretrained model for partitioning and extracting the feature of the multimodal sequential data.”
Wei, T., Chen, H., Liu, W., Chen, L., Gu, P., & Wang, J. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3638382 · Cited by 1
“Modality-share information refers to the common characteristics across modalities [8], [14], such as building shape and spatial location, which can bridge their semantic gap [15].”
Liu, W., Li, T., & Chen, Y. (2025) · Multimedia Systems · doi:10.1007/s00530-025-02088-7 · Cited by 1
Gui, J., Xia, Z., Li, Y., & Chen, J. (2025) · Journal of Food Process Engineering · doi:10.1111/jfpe.70291 · Cited by 1
Sun, H., Lu, X., Tian, J., Zhou, J., He, J., Wang, H., Kong, X., Hu, X., & Qin, Y. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3640882 · Cited by 1
“[21],inspiredbythedomainseparationnetwork[23],incor-porateanorthogonalitylosstostrengthenthecomplementarity betweenmodalitiesandimproveoverallperformance.”
Zhang, H., Zhou, Y., Sun, K., Pang, Y., & Xu, G. (2025) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3743093.3771008 · Cited by 1
“Hazarika et al [4] proposed combining different losses to capture cross-modal commonalities to reduce modal gaps.”
Multimodal Sentiment Analysis with Contrastive Constraint Propagation
Wang, G., & Zhang, M. (2025) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3743093.3771056 · Cited by 1
“Following prior work [5], we adopt comprehensive evaluation metrics including he performance of MSA is evaluated using multiple metrics: 7-class accuracy (Acc-7), 5-class accuracy (Acc-5), binary accuracy (Acc-2), F1 score, correlation between model predictions and human annotations (Corr), and…”
Cui, L., Liu, S., Wang, L., & Yang, S. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229039 · Cited by 1
“For example, MISA [20] and Self-MM [21] replace traditional word embeddings like GloVe [22] with BERT to extract contextualized features from the original sentence.”
Improving Emotion and Intent Understanding in Multimodal Conversations With Progressive Interaction
Zheng, L., Song, T., Ding, Y., Wu, X., Li, F., Xie, D., Li, J., Teng, C., & Ji, D. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3644684 · Cited by 1
“(cid:2) MISA: [48] introduce a mechanism for learning modality invariant and modality specific feature representations.”
Emotion Recognition in Human–Computer Interaction Using Multimodal Adaptive Reweighting Strategy
Li, Y., Ding, Y., Yan, X., Yang, L., & Liu, Y. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3649251 · Cited by 1
Li, J., Lu, X., Li, Z., Liu, X., Liu, H., & Ruan, J. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06345-4 · Cited by 1
Pan, L., & Li, H. (2025) · IEEE Access · doi:10.1109/access.2025.3649994 · Cited by 1
“MISA [5]: By learning modality-invariant and modality-specific representations, it effectively performs representation learning on multimodal data before fusion, thus significantly improving prediction performance in multi-modal sentiment analysis.”
Wang, H., Yu, B., Xi, Z., Zhao, S., & Yang, Y. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3651100 · Cited by 2
“…how to capture modality-invariant and modality-specific aspects, and designing different loss functions and decoupling operations to handle multi-view multimodal data [4], [11], [12], [13], [14], different explorations have also been conducted on modality weight allocation [6], [10], [15], [16].”
Confidence-guided dynamic sequential fusion for multimodal sentiment analysis
Xia, R., Lin, Y., Qin, G., He, L., & Li, Y. (2026) · World Wide Web · doi:10.1007/s11280-025-01401-8 · Cited by 1
A Review of Multimodal Sentiment Analysis in Online Public Opinion Monitoring
Liu, S., & Li, T. (2026) · Informatics · doi:10.3390/informatics13010010 · Cited by 1
G-TBML: GAN-enhanced Transformer-based Multimodal Binding Learning Model
Qu, W., & Wei, B. (2025) · International Conference on Machine Learning and Computer Application (ICMLCA) · doi:10.1109/icmlca66850.2025.11336479 · Cited by 1
“…MFN mode [20] proposed by Zadeh et al., the RAVEN mode [30] by Wang et al., the MulT model [31] and MFM model [32] by Tsai et al., the MISA model[9] by Hazarika et al., the ICCN model [33] by Sun et al., the TETFN model [6] by Wang et al., and the foundational model TMBL [14] utilized in this…”
InfluentialMultimodal Classroom Climate Recognition: Dataset, Method, Application
Lu, W., Chen, Y., Yang, Y., Song, R., Wang, T., & Bian, C. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3647130 · Cited by 1
“MISA [74] introduces modality-specific latent representations to capture unique modality characteristics, enhancing interpretability, while Self-MM [75] combines self-supervised and multi-task learning to improve generalization, particularly for multimodal sentiment analysis.”
Narejo, K. R., Zan, H., Mostafa, S. M., Karim, F. K., Mehmood, F., & Yaseen, A. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00447-y · Cited by 2
A Mixture-of-Experts model for multimodal emotion recognition in conversations
Dutta, S., Balaji, S., & Ganapathy, S. (2026) · Computer Speech & Language · doi:10.1016/j.csl.2026.101965 · Cited by 6
Sun, Z., Liu, H., Li, Y., Li, H., Zhang, W., & Song, T. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00610-z · Cited by 2
“The second line of work first disentangles shared and modality-specific sub-spaces before fusion, and then further optimizes fusion quality using auxiliary constraints, including distribution alignment, independence and orthogonality regularization, and mutual-information-based objectives, as in MISA (Haz-arika et al. 2…”
Multimodal sentiment analysis based on dual perspective modeling
Gulanbaier, T., Hao, W., & Wushouer, M. (2026) · Multimedia Systems · doi:10.1007/s00530-026-02213-0 · Cited by 1
Refined separated representation learning and conflict alleviation for multimodal sentiment analysis
Zheng, Y., Lu, A., Peng, S., Chen, Y., & Peng, X. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00568-y · Cited by 1
Chen, W., Wu, J. Y., Nie, T., Xu, R., & Tu, L. (2025) · International Conference on the Frontiers of Electronic, Electrical and Information Engineering (ICFEEIE) · doi:10.1109/icfeeie66944.2025.00036 · Cited by 1
MIME: Mutual information-guided mixture of experts for robust multimodal sentiment analysis
Zhang, D., Xiao, Q., He, X., & Zhang, J. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00587-9 · Cited by 1
“MISA Hazarika et al. (2020) projects each modality into modality-invariant and modality-specific subspaces, learning both shared and private representations to better capture complementary information.”
InfluentialEvaluating sentiment analysis models in healthcare: addressing bias and enhancing interpretability
Wang, C., Miao, Z., & Zeng, H. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1663871 · Cited by 1
Cao, D., Zhang, J., Chen, Z., & Cheng, H. (2026) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2026.3669462 · Cited by 1
“Hazarika et al. [19] devised six modality representations, including three modality-invariant representations and three modality-specific representations.”
Zheng, N., Tang, Y., Chen, J., Liu, H., & Chen, Q. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00671-0 · Cited by 1
“Early studies (Han et al. 2021; Hazarika et al. 2020; Li et al. 2023; Lu et al. 2025; Wang et al. 2025; Wei et al. 2023; Wu et al. 2024; Xie et al. 2024; Yu et al. 2021; Zhang et al. 2022) primarily focused on developing advanced fusion strategies and interaction mechanisms to improve sentiment…”
Huang, J., Zhu, D., Zitong, Y. U., Ma, H., Zhang, J., Zhu, C., & Cao, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37447-9_33 · Cited by 1
CCAF: Coarse-to-fine Cross-Modal Alignment and Fusion for Multimodal Sentiment Analysis
Zhao, X., Yang, S., & Tang, B. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792569 · Cited by 1
A contrastive graph-transformer network model for multimodal sentiment analysis
Do, H. N., Phan, H. T., & Nguyên, N. T. (2026) · Applied Intelligence · doi:10.1007/s10489-026-07235-9 · Cited by 1
Hybrid Progressive Fusion Network for Multimodal Sentiment Analysis
Li, X., Leng, J., Cheng, X., Wang, T., & Zhang, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461705 · Cited by 1
GCR-TE: Grounded generative counterfactual rewriting for text enhanced multimodal sentiment analysis
Ji, S., & Yang, B. (2026) · Array · doi:10.1016/j.array.2026.100849 · Cited by 1
Wang, Z., 姚朱伊, Shen, J., Wang, P., Chen, X., Zhang, X., Gu, H., & Zhou, X. (2026) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2026.3681252 · Cited by 1
“Hazarika et al. [11] proposed MISA, which disentangles modality-invariant and modality-specific factors via adversarial learning to improve generalization.”
Zhuang, Y., Liu, M., Bai, W., Zhang, Y., Zhang, X., Deng, J., & Ren, F. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00440 · Cited by 3
“Here, we use a transformer encoder [36] followed by a linear layer [10] as the F w for fusion.”
BIG-TM: Bridging Individual Guidance with Trifusion MoPoE for Chinese memes understanding
Wang, T., Li, J., Xue, F., Su, D., Su, G., Wen, X., & Sha, Y. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116137 · Cited by 1
Zhang, Y., Kuang, W., Huang, Z., Fan, C., & Zou, Y. (2026) · IEEE International Conference on Pervasive Computing and Communications · doi:10.1109/percom67906.2026.11524348 · Cited by 1
Dynamic cross-instance context mining for multimodal sentiment analysis
Xia, H., Cheng, Y., Tan, Y., & Song, S. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104921 · Cited by 1
A mamba-driven transformer method for multimodal emotion recognition optimization
Ren, D., & Guo, S. (2026) · Alexandria Engineering Journal · doi:10.1016/j.aej.2026.04.062 · Cited by 1
Dynamic Interaction-Aware and Causality-Disentangled Framework for Multimodal Sentiment Analysis
Jian, H., Chen, D. F., Lei, B., & Jinhua, S. (2026) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit70033.2026.11557593 · Cited by 1
“The learning procedures are delineated as follows: ( v x Transformer U (cid:32) (3) For subsequent computations, auditory and visual features are projected into the same fixed dimensionality as text using a fully connected layer, and the input for all modalities can be denoted as Following the…”
HDR: Hypergraph dynamic representation network for multimodal sentiment analysis
Xiao, K., Zhang, C., & Pang, S. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133313 · Cited by 1
Causal mixture-of-experts for robust multimodal fusion
Yang, L., Long, X., Zeng, Y., Chen, H., Xiao, Y., & Yang, J. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114306 · Cited by 1
GICA: gate-controlled information bottleneck cross-modal attention network
Qiang, Y., & Shen, J. (2026) · Pattern Analysis and Applications · doi:10.1007/s10044-026-01731-x · Cited by 1
Рюмина, Е. В., Axyonov, A., Koryakovskaya, D., Abdulkadirov, T., Egorova, A., Fedchin, S., Zaburdaev, A., Salova, I., Savchenko, A., & Ryumin, D. (2026) · IEEE Access · doi:10.1109/access.2026.3723020 · Cited by 1
“Hazarika et al. [16] proposed a Transformer-based network in which features from each modality are projected into two distinct sub-spaces: modality-invariant and modality-specific.”
Zhang, W., Ding, J., Liu, H., Han, T., Liu, Y., & Yang, L. T. (2026) · Big Data Mining and Analytics · doi:10.26599/bdma.2025.9020110 · Cited by 1
Che, X., Wang, W., Guan, J., & Zhao, Q. (2026) · International Conference on Computer Information Science and Application Technology (CISAT) · doi:10.1109/cisat71030.2026.11682622 · Cited by 1
Label-Aware Entropic Distributional Contrastive Alignment for Multimodal Sentiment Analysis
Wang, M., Meng, X., & Wang, C. (2026) · Entropy · doi:10.3390/e28101062 · Cited by 1
SLBERT: A Novel Pre-Training Framework for Joint Speech and Language Modeling
Susladkar, O., Gatti, P., & Yadav, S. K. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094658 · Cited by 1
“A prediction for an utterance is correct only if all slot Graph-MFN [3] 76.90% CAE-LR [21] 78.00% Gated Multimodal Attention Fusion [22] 81.14% Multilogue-Net [19] 82.10% MMLatch [23] 82.40% Transformer-based Joint Modeling [20] 82.40% MISA [24] 83.60% MAG-XLNet [25] 85 1.”
MethodInfluentialZhang, Q., Shi, L., Liu, P., & Xu, L. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191402 · Cited by 1
“For example, Tensor Fusion Network[4] corrects for the spatial orientation of unimodal or multimodal representations by calculating the multidimensional tensor of the modality based on the outer product; by matrix decomposition[5] or by Euclidean metric optimization[6, 7] to obtain optimized fusion information.”
BackgroundMethodPSRMER: Proactive Services Recommendation Driven-by Multimodal Emotion Recognition
Liu, Z., Huang, G., Chu, D., & Sun, Y. (2023) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws60048.2023.00070 · Cited by 1
“• MISA [40]: The invariant and specific representation of modalities projects the elements into two individual spaces with special constraints, and then fusion is applied to these modal features.”
Exploiting Multi-modal Fusion for Robust Face Representation Learning with Missing Modality
Zhu, Y., Xin, S., & Zhou, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44210-0_23 · Cited by 1
Key Factor Analysis of Influencer Popularity based on Adjective and Emotion in Twitter User Opinions
Masuda, S., & Yamasaki, T. (2023) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat59888.2023.00041 · Cited by 1
“It has applications in diverse domains, including customer reviews, brand perception, social media opinion analysis and political sentiment surveys [2], [3], [7]–[10].”
BackgroundMiao, Y., Zhu, G., Liu, T., Zhang, W., Wen, Y., & Zhou, M. Q. (2023) · International Symposium on Parallel Architectures, Algorithms and Programming · doi:10.1109/paap60200.2023.10391354 · Cited by 1
“MISA [19] :learns modal invariance and specific representations by projecting each sample modality into two subspaces.”
BackgroundHan, X., Chen, F., & Ban, J. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-0576-4_4 · Cited by 1
Multimodal Sentiment Analysis Based on Information Bottleneck and Attention Mechanisms
Chen, X., Wu, Z., Tang, Y., & Han, R. D. (2023) · International Conference on Cloud Computing, Big Data Application and Software Engineering (CBASE) · doi:10.1109/cbase60015.2023.10439124 · Cited by 1
“…the information from the remaining two modes is transferred to the target mode, and the features of the modes are fully extracted 5) MISA [27]:Learning effective modal representations to aid fusion, continuously optimizing model parameters to learn semantic information about multimodal…”
MethodMSA-HCL: Multimodal sentiment analysis model with hybrid contrastive learning
Zhao, W., Zhang, Y., Hua, Q., Dong, C.-R., Wang, J., & Zhang, F. (2024) · Mathematical Foundations of Computing · doi:10.3934/mfc.2024017 · Cited by 1
Attention-Based Fusion of Intra- and Intermodal Dynamics in Multimodal Sentiment Analysis
Yaghoubi, E., Tran, T. K., Borza, D. L., & Frintrop, S. (2024) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops59983.2024.10502594 · Cited by 1
Progressive Fusion Network with Mixture of Experts for Multimodal Sentiment Analysis
Liu, D., Yang, Z., & Guo, Z. (2024) · International Conference on Advanced Computational Intelligence (ICACI) · doi:10.1109/icaci60820.2024.10537022 · Cited by 1
Multi-Feature Fusion Multi-Modal Sentiment Analysis Model Based on Cross-Attention Mechanism
Pan, Z., Zhao, L., & Tan, Y. (2024) · International Conference on Cloud Computing and Big Data Analytics (ICCCBDA) · doi:10.1109/icccbda61447.2024.10569861 · Cited by 1
“Hazarika et al. [9] proposed the MISA framework, which maps each mode to private and shared spaces across modes, achieving a combination of internal and intermodal interactions.”
BackgroundResultInfluentialMultimodal sentiment analysis based on TCN and cross-modal interactive feedback network
Bao, G., Shen, Z., Liu, C., Sun, L., & Chen, S. (2024) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd61410.2024.10580733 · Cited by 1
MCAF: Multi-Headed Causal Attention Fusion Approach for Multimodal Sentiment Analysis
Wang, Q., Wang, X., Yang, J., & Bi, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650874 · Cited by 1
“With earlier adopted multimodal fusion methods such as MFM [28], MulT [29], MISA [30], ICCN [31], and Self-MM [32].”
MethodSun, L., Zhao, C., Li, X., Bai, C., & Pan, J. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651112 · Cited by 1
“To objectively illustrate the effectiveness of our model, we compare it with the following baselines and state-of-the-art models, include TFN [7], LMF [8], MulT [34], MISA [35], MAG-BERT [36], Self-MM [26], AMML [29], MUTA-Net [28], ConFEDE [37].”
MethodMa, T.-F., Liu, L., Zhao, L.-M., Peng, D., Lu, Y., Zheng, W., & Lu, B. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650558 · Cited by 2
“The fusion part employs the modality-invariant and specific representation structure [38].”
BackgroundShi, L., Xu, F., Wang, R., Wei, Y., Wang, G., Wang, B., & Liu, P. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10687981 · Cited by 1
“Some studies suggest that fusion various modalities can learn complementary informations, leading to improved joint multimodal representations [2]–[6].”
BackgroundCH-Mits: A Cross-Modal Dataset for User Sentiment Analysis on Chinese Social Media
Ma, J., Xu, S., Liu, Y., & Fu, X. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679125 · Cited by 1
“(2) Self-MM [8] 95.06 95.35 94.74 95.15 Self-MM [24] 95.80 96.00 95.58 95.65 MISA [9] 96.05 96.21 95.88 95.96 HFN [17] 97.04 97.12 96.95 97.20 MAG-BERT [18] 97.53 97.61 97.45 97.59 PEM 𝑇𝐹𝑁 [25] 96.79 96.74 96.84 97.04 PEM 𝐿𝑀𝐹 [14] 96.79 96.48 97.05 97.12 PEM 𝑃𝑇𝑃 [10] 97.53 [25] is the…”
InfluentialUncertainty-Aware Gradient Modulation and Feature Masking for Multimodal Sentiment Analysis
Wu, Y., Wang, C., Li, J., Zhang, W., & Jiang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8795-1_22 · Cited by 1
Wang, X., Wang, M., Cui, H. N., & Zhang, Y. (2024) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3696409.3700255 · Cited by 1
“We compared MLBM with widely recognized models in seven domains, including LMF [14], TFN [28], MulT [15], BBFN [5], TextMI [6], MISA [8], and I-Code [27], to verify the advantages of our proposed model.”
MFM: Multimodal Sentiment Analysis Based on Modal Focusing Model
Sun, S., Xu, G., & Lu, S. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10831023 · Cited by 1
Katada, S., Okada, S., & Komatani, K. (2025) · Transactions of the Japanese Society for Artificial Intelligence · doi:10.1527/tjsai.40-2_b-o72 · Cited by 1
A unified prompt-based framework for few-shot multimodal language analysis
Zhang, X., Cao, R., Wang, Y., Li, S., Xu, H., Gao, K., & Huang, L. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200498 · Cited by 1
Commonality Augmented Disentanglement for Multimodal Crowdfunding Success Prediction
Li, J., Xu, X., Li, Y., Zhong, T., Zhang, K., & Zhou, F. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889564 · Cited by 1
“For instance, in [14] the researchers disentangles the multimodal data into modality-invariant and modality-specific space, corresponding to synergetic (commonality) and complementary (specificity) effects, respectively.”
Zhang, T., Tan, J., Xiong, Z., Wu, B., & Zheng, C. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888426 · Cited by 1
“To demonstrate the effectiveness of CAMAL, we compare it with several widely used DNNs: LMF [34], MISA [35], Self-MM [36], CubeMLP [37], LAD [1], and CLUE [23].”
Convolutional channel modulator for transformer and LSTM networks in EEG-based emotion recognition
Kang, H. W., Choi, J. W., & Kim, B. H. (2025) · Biomedical Engineering Letters · doi:10.1007/s13534-025-00475-7 · Cited by 1
A Multimodal, Multilabel Approach to Recognize Emotions in Oral History Interviews
Viswanath, A., Gref, M., Hassan, T., & Schmidt, C. (2024) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii63134.2024.00038 · Cited by 1
“…categories, such as the Convolutional Attention Network (ConvNet) [52], Modality-Invariant and Specific Representations (MISA) model [53], Modality-transferable model (MTM) [54], or based on other novel multimodal fusion techniques such as the Low-rank Multimodal Fusion (LMF) model…”
Prototype-oriented multimodal emotion contrast-enhancer
Zhang, Q., Yang, Q., Tian, S., Yu, L., Fan, X., & Song, J. (2025) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2025.110393 · Cited by 1
Shuffling Augmented Decoupled Features for Multimodal Emotion Recognition
Cho, S. (2025) · IEEE Access · doi:10.1109/access.2025.3572925 · Cited by 2
“We compare our method with the following MER approaches: TFN [7], LMF [8], MFM [9], ICCN [10], MISA [11], MulT [5], MAG-BERT [16], HyCon [17], DMD [12], MMER-TD [59], and ALMT [60].”
InfluentialSGMN: Self-Supervised Guidance for Modality-Specific Network in Multimodal Sentiment Analysis
Lu, B., & Li, T. (2025) · International Conference on Neural Networks, Information and Communication Engineering (NNICE) · doi:10.1109/nnice64954.2025.11064322 · Cited by 1
“In alignment with the research conducted by Hazarika et al. [1], our experimental outcomes are assessed using two distinct formats: multi-class classification and regression.”
Unified-Modality Attention Network for Multimodal Sentiment Analysis
Li, Z., & Li, L. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11210163 · Cited by 2
Wang, J.-F., Tsai, H., & Liu, J.-M. (2025) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001425510231 · Cited by 1
Neural Dependency Coding inspired Multimodal FusionPreprint
Shankar, S. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.00385 · Cited by 3
Pedestrian Re-Identification Based on Fine-Grained Features Learning and FusionPreprint
Chen, A., Lin, Q., & Liu, W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4879811 · Cited by 1
A Multi-step Attention and Multi-level Structure Network for Multimodal Sentiment Analysis
Zhang, C., Zhao, H., Wang, B., Wang, W., Ke, T., & Li, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17120-8_56
Capturing Cross-View Dynamics Using Recurrent Neural Networks for Multi-modal Sentiment Analysis
Chitale, P., Dhope, T., & Kalbande, D. R. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-8477-8_20
Multi-Scale Compositional Constraints for Representation Learning on Videos
Paraskevopoulos, G., Lavania, C., Chum, L., & Sundaram, S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096573
“Multimodal representations improve performance on several real world tasks, e.g. action recognition on videos [1, 2], sentiment analysis [3, 4], or speech recognition [5, 6].”
BackgroundWang, Y., Guo, X., Zhang, Y., Ren, Y., Huang, W., Liu, Z., Feng, Y., Dai, X., Zhang, W., & Che, H. (2023) · International Conference on Advanced Computational Intelligence (ICACI) · doi:10.1109/icaci58115.2023.10146184
“The core approach of MISA lies in the use of loss functions, similarity loss ( )sL to reduce the differences in common information between modalities, and difference loss ( )dL to capture different aspects of the input features and then fuse these modal representations, using the features in the shared subspace of the…”
MethodInfluentialAn adversarial training-based mutual information constraint method
Liu, R., Zhang, X., Wang, J., & Zhou, X. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04803-1
Modal Interactive Feature Encoder for Multimodal Sentiment Analysis
Zhao, X., Zhou, J., & Xiu-juan, X. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-5971-6_21
Cross-Modal Sentiment Analysis
Xu, H. (2023) · Springer eBooks · doi:10.1007/978-981-99-5776-7_5
Multimodal Sentiment Analysis Data Sets and Preprocessing
Xu, H. (2023) · Springer eBooks · doi:10.1007/978-981-99-5776-7_2
Multimodal Sentiment Analysis Platform and Application
Xu, H. (2023) · Springer eBooks · doi:10.1007/978-981-99-5776-7_7
Xu, H. (2023) · Springer eBooks · doi:10.1007/978-981-99-5776-7_1
MetaSelection: A Learnable Masked AutoEncoder for Multimodal Sentiment Feature Selection
Liang, X., Chen, H., Xuan, H., & Zhou, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8540-1_14
Hai, J., Liu, Y.-J., & Elhilali, M. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389694
Adaptive Token Selection and Fusion Network for Multimodal Sentiment Analysis
Li, X., Lu, M., Guo, Z., & Zhang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-53311-2_17
Li, Y., Huang, J.-X., Wang, X.-Y., Pei, X., & Lu, Y. (2024) · Medical Imaging · doi:10.1117/12.3005603
“26,27 Let X be a dataset batch of N patients, where each X = [ x 1 , x 2 ] contains data from SWE and BUS.”
MethodLAMB: Label-Induced Mixed-Level Blending for Multimodal Multi-label Emotion Detection
Qian, S., Guo, M., Fan, Z., Chen, M., & Wang, C. (2024) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-54528-3_2
SWACL: Sentimental Words Aware Contrastive Learning for Multimodal Sentiment Analysis
Zhuang, J., He, J., & Li, X. (2023) · International Conference on Computing and Pattern Recognition (ICCPR) · doi:10.1145/3633637.3633638
MD-EPN: An Efficient Multimodal Emotion Recognition Method Based on Multi-Dimensional Feature Fusion
Xie, K., & Xie, Z. (2024) · International Conference on Neural Networks, Information and Communication Engineering (NNICE) · doi:10.1109/nnice61279.2024.10498191
“MISA[19] solely relies on the BERT pre-trained model to extract audio features and segregates each modality into modality-invariant and modality-specific features.”
MethodMulti-token Fusion Framework for Multimodal Sentiment Analysis
Long, Z., Deng, H., Yang, Z., & Liu, W. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2390-4_29
Tri-Modalities Fusion for Multimodal Sentiment Analysis
Wang, H., Liu, S., Zhao, D., Chen, Y., & Ji, W. (2023) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait60137.2023.10528533
“Previous studies in this field [14], [15], [18] also used triple symmetric architectures in which the bidirectional relationships in each modal pair are modeled in some way.”
Background
Metadata
- Authors
- Hazarika, D., Zimmermann, R., & Poria, S.
- Affiliations
- National University of Singapore (SG)
- Singapore University of Technology and Design (SG)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 1122-1131
- OpenAlex
- W3093051361
- Cited by
- 1,073
- References
- 45
- Citations by year
- 2026: 280 · 2025: 315 · 2024: 238 · 2023: 136 · 2022: 78 · 2021: 23 · 2020: 3
- FWCI
- 22.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- Funders
- Ministry of Education - Singapore (MOE2018-T2-1-103)
- DSO National Laboratories - Singapore (RTDST190702)
- RIE 2020 Advanced Manufacturing and Engineering (AME) programmatic grant (A19E2b0098)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, modality gap, modality-invariant representations, modality-specific representations, subspace projection, multimodal fusion, user-generated video, humor detection, MOSI, MOSEI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
