The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Open accessAbstract
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On CMU-MOSEI dataset, our proposed C-MIB yields 0.7% improvement on Acc2 and 1.2% improvement on F1 score compared to the current state-of-the-art MISA [53]. The performance on Acc7 and Corr is also improved compared to the state-of-the-art methods.”
From Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations (2023) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 8×
“Inter-utterance context. et al. proposed one of the first models, bc-LSTM , which utilized”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“However, following recent works [7], including the stateof-the-art ICCN [50], we utilize the pre-trained BERT [11] as the feature extractor for textual utterances.”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“ndard low-level features that are provided by the respective benchmarks and utilized by the state-of-the-art methods. 4.3.1 Language Features. Traditionally, language modality features has been GloVe [38] embeddings for each token in the utterance. However, following recent works [7], including the stateof-the-art ICC…”
MethodInfluential(2008) · Journal of Machine Learning Research · Cited by 33,051
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 5×
“licly available models of [26, 51, 58]).”
BackgroundInfluentialSelf-citationEkman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,971 · cited 1×
“Both MOSI and MOSEI use Facet 3 to extract facial expression features, which include facial action units and face pose based on the Facial Action Coding System (FACS) [14].”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · cited 1×
“For UR_FUNNY, OpenFace [4], a facial behavioral analysis tool, is used to extract features related to the facial expressions of the speaker.”
Method
Research works printed but not linked: 15
- Andrew Galen, 2013, Proceedings of the 30th International Conference on Machine Learning, ICML 2013 (JMLR Workshop and Conference Proceedings, 1255
- Bousmalis Konstantinos, Advances in Neural Information Processing Systems 29. Curran Associates
- 10.5555/2354409.2355082 · DOI
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 19
- Drugman Thomas, 2011, Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonics. In INTERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association. ISCA
- Kiela Douwe, 2019, NeurIPS 2019 Workshop. Curran Associates, Inc.
- Lu Jiasen, Advances in Neural Information Processing Systems 32. Curran Associates
- 10.5555/2392963.2392965 · DOI
- Peng Minlong, 2018, Long Papers
- Rajagopalan Shyam Sundar, 2016, Proceedings, Part VII (Lecture Notes in Computer Science, 353
- Salzmann Mathieu, 2010, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010 (JMLR Proceedings, 708
- Hubert Tsai Yao-Hung, 2019, Learning Factorized Multimodal Representations. In 7th International Conference on Learning Representations, ICLR
- Vaswani Ashish, Advances in Neural Information Processing Systems 30. Curran Associates
- Zadeh Amir, 2018, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, 2246
- Zellinger Werner, 2017, Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning. In 5th International Conference on Learning Representations, ICLR
Cited by
Sorted by citations
Items 401–500 of 1,001
DFNM: Dynamic Fusion Network of Intra- and Inter-modalities for Multimodal Sentiment Analysis
Li, Z., Ma, J., Li, X., & Pan, X. (2021) · International Conference on Asian Language Processing · doi:10.1109/ialp54817.2021.9675255 · Cited by 5
Unified Multi-Modal Multi-Task Joint Learning for Language-Vision Relation Inference
Lu, W., & Zhang, D. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859683 · Cited by 5
“Multi-modal analysis with a focus on language and vision has become a very hot topic in natural language processing, computer vision, and multimedia understanding [1–4].”
BackgroundER-MRL: Emotion Recognition based on Multimodal Representation Learning
Guo, X., Wang, Y., Miao, Z., Yang, X., Guo, J., Hou, X., & Zao, F. (2022) · International Conference on Information Science and Technology · doi:10.1109/icist55546.2022.9926848 · Cited by 5
Research on cross-modal emotion recognition based on multi-layer semantic fusion
Xu, Z., & Gao, Y. (2024) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2024110 · Cited by 5
A Review of Multimodal Sentiment Analysis: Modal Fusion and Representation
Fu, H., & Lu, H. (2024) · International Wireless Communications and Mobile Computing Conference (IWCMC) · doi:10.1109/iwcmc61514.2024.10592484 · Cited by 6
“Currently, modal differences may be quantified using techniques like classification loss [42] and geometric measurement [43], and the difference can be utilized as the regularization term in the final loss.”
MethodTemporal Enhancement for Video Affective Content Analysis
Li, X., Wang, S., & Huang, X. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681631 · Cited by 5
Nie, X., Yang, L. T., Li, Z., Deng, X., Fan, F., & Yang, Z. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3459929 · Cited by 6
Dong, Z., Wang, D., Chen, C., Huang, D., & Zhang, Z. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887877 · Cited by 5
“For multimodal fusion, we used mainstream strategies like TFN [39], LMF [40], and MISA [41] at the utterance level, and methods like MFN [42] and MulT [43] at the sequence level.”
A Novel Decoupled Prototype Completion Network for Incomplete Multimodal Emotion Recognition
Hu, Z., Zheng, W., Zong, Y., Wei, M., Jiang, X., & Shi, M. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10687673 · Cited by 6
Cai, T., Wang, S., Wang, J., Xiong, Y., & Liu, L. (2025) · Frontiers of digital education. · doi:10.1007/s44366-025-0063-x · Cited by 5
“In this case, facial expression, voice intonation, and lecture text come from the same video segment and should have the same emotional polarity, such as emotion commonality (Hazarika et al., 2020).”
Ouyang, X., & Wei, J. (2025) · Annals of Biomedical Engineering · doi:10.1007/s10439-025-03865-1 · Cited by 16
Jiang, S., Wang, Z., Wu, X., Ji, H., Li, M. X., Li, X., & Gao, C. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3754817 · Cited by 6
Enhanced sentiment analysis in tourism reviews via multimodal graph convolutional networks
Yang, W.-J., Xue, Z., Fu, G.-T., Chen, G., & Yang, C. (2025) · PLoS ONE · doi:10.1371/journal.pone.0333628 · Cited by 3
“Paper [29] focused on learning modality-invariant and -specific representations for multimodal sentiment analysis to facilitate the fusion process.”
Bridging modal gaps in multimodal sentiment analysis: a self-supervised text-guided fusion approach
Zhong, H., Yi, J., Hu, R., Chen, A., & Han, G. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.131012 · Cited by 4
Decoding Multimodal Text Analytics: Tasks, Datasets, Fusion Models, and Future Frontiers
Nath, T., Gupta, V., Gupta, M., & Sharma, R. R. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70083 · Cited by 4
MoWER: Multimodal emotion recognition in conversation with emotion wheel-enhanced representation
He, J., Cai, J., Yang, M., Xiao, Y., Cai, L., & Kang, Y. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116044 · Cited by 4
Zhao, X., Miao, X., Xu, X., Liu, Y., & Cao, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650083 · Cited by 4
Multi-Hop Attention Diffusion Graph Neural Networks For Multimodal Fake News Detection
Huang, Z., Lu, D., & Sha, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887755 · Cited by 4
“Second, although multimodal information facilitates fake news detection, there are modality gaps that persist between heterogeneous modalities [15].”
Decouple and Rebalance: Towards effective joint learning for EEG-fNIRS integration
Meng, M., Hao, D., Gao, Y., Li, L., Wang, M., & Xi, X. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130128 · Cited by 4
Learning by Comparing: Boosting Multimodal Affective Computing through Ordinal Learning
Mai, S., Zeng, Y., & Hu, H. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714841 · Cited by 4
RCRank: Multimodal Ranking of Root Causes of Slow Queries in Cloud Database Systems
Ouyang, B., Zhang, Y., Cheng, H., Yang, S., Guo, C., Yang, B., Wen, Q., Fan, L., & Jensen, C. S. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3717755.3717774 · Cited by 4
“To study the impact of different 𝜆 on performance, we investigate the average performance of the Hologres datasets and the performance of the TPC-DS dataset with values of 𝜆 in [ 1 , 3 , 5 , 7 , 10 ] .”
A Multimodal Sentiment Analysis Approach Based on Multiview Cross-Modal Fusion
Zhi, Y., Li, J., Wang, H., & Chen, J. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3583711 · Cited by 5
“In terms of the means of implementation of sentiment analysis, video-based sentiment recognition can be accomplished by unimodal [10], dual-modal [11] and tri-modal [12], [13], [14], [15].”
InfluentialWang, X., Xie, L., Li, C., Wang, M., Liu, Z., Peng, X., & Wang, Z. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01925-z · Cited by 4
$$A^{2} H^{2}$$ for multimodal emotional data analysis
Wu, J., Liu, J., Zhang, T., Guo, S., Chen, Y., Huang, J., & Deng, F. A. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00974-0 · Cited by 4
Intra-Sample and Intra-Modal Enhancement for Multimodal Sentiment Analysis With Missing Modalities
Yan, Z., Zhang, Y., Deng, J., & Ren, F. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3645559 · Cited by 4
“…using cross-modal attention [4], [7], [13], [14], some methods decouple representations into modality-specific and modality-common components [15], [16], [17], while others utilize information-theoretic approaches, such as mutual information [6], [18], [19], [20], to enhance cross-modal…”
Accurate Hand Gesture Recognition Using Color and Depth Images with Modality-invariant Fusion
Okano, M., Liu, J., Tateyama, T., Iwamoto, Y., & Chen, Y. (2022) · IEEE Global Conference on Consumer Electronics (GCCE) · doi:10.1109/gcce56475.2022.10014261 · Cited by 4
“A similarity loss was added [7] to align the feature distribution gap between the color and depth images.”
MethodHierarchical Supervised Contrastive Learning for Multimodal Sentiment Analysis
Chen, K., Wang, S., & Hao, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-53308-2_5 · Cited by 4
Teng, S., Jiaqing, L. I. U., HUANG, Y., Chai, S., Tateyama, T., Huang, X., Lin, L., & Chen, Y. (2024) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2023hcp0006 · Cited by 5
Conditional selection with CNN augmented transformer for multimodal affective analysis
Wang, J., Wang, S., Xiao, S., Lin, R., Dong, M., & Guo, W. (2024) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12320 · Cited by 5
Ma, L., Li, J., Shao, D., Yan, J., Wang, J., & Yan, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3405586 · Cited by 4
“Building on the methodology of previous studies [12], [13], [21], we present our experimental findings in two distinct categories: classification and regression.”
BackgroundMethodFEF-Net: feature enhanced fusion network with crossmodal attention for multimodal humor prediction
Gao, P., Tao, C., & Guan, D. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01402-z · Cited by 4
QuMIN: quantum multi-modal data fusion for humor detection
Phukan, A., Khan, A. A. H., & Ekbal, A. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19790-9 · Cited by 4
A Fine-Grained Sentiment Analysis Method Using Transformer for Weibo Comment Text
Xue, P., & Bai, W. (2024) · International Journal of Information Technologies and Systems Approach · doi:10.4018/ijitsa.345397 · Cited by 4
MultiHGR: Multi-Task Hand Gesture Recognition with Cross-Modal Wrist-Worn Devices
Lyu, M., Zhou, H., Guo, K., Zhou, W., Shen, X., & Gu, Y. (2024) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom52122.2024.10621430 · Cited by 4
SmartRAN: Smart Routing Attention Network for multimodal sentiment analysis
Guo, X., Tian, S., Yu, L., & He, X. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05839-7 · Cited by 4
Multi-modal Intent Detection with LVAMoE: the Language-Visual-Audio Mixture of Experts
Li, T., Bao, J., Qin, J., Liang, Y., Zhang, R., & Wang, J. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10688018 · Cited by 4
“Inspired by the research of Hazarika [7], we use two different approaches to learn modality-invariant and modality-specific feature representations.”
MethodDual-View Multimodal Interaction in Multimodal Sentiment Analysis
Zhu, K., Fan, C., Tao, J., Xue, J., Xie, H., Liu, X., Li, Y., Wen, Z., & Lv, Z. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10688078 · Cited by 4
Meng, J., Zhu, Z., Qi, J., & Zhang, H. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06524-9 · Cited by 4
Fan, Q., Zuo, H., Liu, R., Lian, Z., & Gao, G. (2024) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3689092.3689411 · Cited by 7
“In the research of MER, remarkable performance depends heavily on the complete multimodal data and robust joint representation learning [15].”
BackgroundFang, J.-F., Wang, W., Lin, G., & Lv, F. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680703 · Cited by 4
Gated Fusion Network With Progressive Modality Enhancement for Sentiment Analysis
Feng, X., Gong, W., Shi, T., Jing, Z., Li, K., & Gonzàlez, J. (2024) · IEEE Access · doi:10.1109/access.2024.3490772 · Cited by 4
Semantic-Driven Crossmodal Fusion for Multimodal Sentiment Analysis
Liu, P., Wang, Z., & Huang, F. J. (2024) · International Journal on Semantic Web and Information Systems · doi:10.4018/ijswis.359985 · Cited by 4
Text-dominant strategy for multistage optimized modality fusion in multimodal sentiment analysis
Wu, J., Wang, J., Jing, S., Liu, J., Zhang, T., Han, M., Zhan, P., & Zuo, G. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01518-2 · Cited by 4
Enhancing emotion recognition through brain asymmetry and multi-angle fusion network
Zhou, B., Lin, L., & Chen, J. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107324 · Cited by 4
SKE-MSA: Enhancing Representation Learning with VAD Lexicon for Multimodal Sentiment Analysis
Yang, Y., Dong, X., Qiang, Y., & Si, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890333 · Cited by 4
“We compared the performance of SKE-MSA with state-of-the-art baseline models, including MFN [21], MulT [22], MISA [23], Self-MM [16], CENET [24], and TETFN [25].”
Chen, G., Qian, Z., Qiu, S., Zhang, D., & Zhou, R. (2025) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105322 · Cited by 5
P2SL: Private-Shared Subspaces Learning for Affective Video Content Analysis
Wang, Q., Xiang, X., Zhao, J., & Deng, X. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859902 · Cited by 4
“Motivated by recent advances [4] in multimodal sentinment analysis, we propose P2SL to learn two distinct representations for each modality.”
BackgroundZhang, T., Dong, C., Su, J., Zhang, H., & Li, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17120-8_44 · Cited by 4
Li, K., Tian, S., Yu, L., Zhou, T., Wang, B., & Wang, F. (2023) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-222189 · Cited by 4
Yang, Z., Li, Z., Zhu, D., & Zhou, Y. (2024) · Information Technology And Control · doi:10.5755/j01.itc.53.1.35060 · Cited by 4
Extracting Multimodal Embeddings via Supervised Contrastive Learning for Psychological Screening
Kalanadhabhatta, M., Santana, A. M., Ganesan, D., Rahman, T., & Grabell, A. S. (2022) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii55700.2022.9953836 · Cited by 4
“[16] projected representations of each modality onto modality-invariant and -specific subspaces and fused them to obtain sentiment predictions.”
BackgroundZero-Shot Visual Emotion Recognition by Exploiting BERT
Kang, H.-W., Hazarika, D., Kim, D., & Kim, J. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-16078-3_33 · Cited by 4
Self-citationFine-Grained Emotion Comprehension: Semisupervised Multimodal Emotion and Intensity Recognition
Fang, Z., Liu, Z., Liu, T., & Hung, C. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3475511 · Cited by 3
“Regarding feature learning methods, Hazarika et al. [22] project each modality’s data into two distinct feature spaces and employ different metric losses to learn modality-independent and modality-specific features.”
An effective Multi-Modality Feature Synergy and Feature Enhancer for multimodal intent recognition
Xia, Y., Song, J., Tian, S., Yang, Q., Fan, X., & Zhu, Z. (2025) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2025.110301 · Cited by 4
Multilevel Representation Disentanglement Framework for Multimodal Sentiment Analysis
Jia, N., Bai, Z., Xiong, T., & Guo, M. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3562827 · Cited by 3
“MISA [25] projected each modality into modality-invariant and modality-exclusive subspaces.”
TsAFN: A two-stage adaptive fusion network for multimodal sentiment analysis
Liu, J., Wang, Y., Yang, J. B., Yu, X., & Zhao, M. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06577-0 · Cited by 3
Multi-modal sentiment analysis based on multi-level modal information interaction
Wang, H., Tuerhong, G., Wushouer, M., & Guo, X. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.08.006 · Cited by 3
Feng, Z., & Giblin, M. M. (2025) · Scientific Reports · doi:10.1038/s41598-025-18665-9 · Cited by 4
LDW: Label Divergence Weighting for Multimodal Sentiment Analysis
Du, Q., Langhe, L. D., Lefever, E., & Hoste, V. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3758160 · Cited by 3
A review of multimodal sentiment analysis: Taxonomy, issues, challenges, and future perspectives
Anwar, K., Shreya, Sharma, M., & Saanvi, K. (2026) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2026.110959 · Cited by 3
Visual Label Augmentation-Driven Multimodal Emotion Recognition
Wei, Q., Zhou, Y., Zhou, J., Ye, L., & Zhang, Y. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113886 · Cited by 3
Translation-based multimodal learning: a survey
Lu, Z., Liao, Y., & Li, J. (2025) · Intelligence & Robotics · doi:10.20517/ir.2025.40 · Cited by 4
Kim, J., & Kim, J. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892785 · Cited by 4
“However, we find that while MISA mapped the representations to the common embedding space, non-linguistic modalities are not well identified.”
BackgroundMethodInfluentialIIMH: Intention Identification In Multimodal Human Utterances
G, K. K. T., DHAKATE, H., & Koolagudi, S. G. (2023) · International Conference on Contemporary Computing · doi:10.1145/3607947.3608016 · Cited by 3
“developed a multimodal sentiment analysis approach in [10], where each modality is projected to two subspaces.”
BackgroundParameter-Efficient Transfer Learning for Audio-Visual-Language Tasks
Liu, H., Xie, X., Gao, Y., & Zhou, Y. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611939 · Cited by 3
Dynamic interactive learning network for audio-visual event localization
Chen, J., Liang, H., Wang, R., Zeng, J., & Lü, P. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05146-7 · Cited by 3
Lightweight Multimodal Fusion Computing Model for Emotional Streaming in Edge Platform
Karjee, J., Kakwani, K., Anand, K., & Praveen Naik, S. (2024) · IEEE Consumer Communications and Networking Conference · doi:10.1109/ccnc51664.2024.10454758 · Cited by 3
“In [10] authors proposed a framework called MISA that factorizes modalities into modality-invariant and modality-specific features where they represent each modality into subspace and later fuse them for sentiment prediction.”
BackgroundGM2RC: Graph-based Multitask Modality Refinement and Complement for Multimodal Sentiment Analysis
Shi, J., Chen, Y., Zhou, S., & Li, M. (2024) · International Symposium on Autonomous Systems (ISAS) · doi:10.1109/isas61044.2024.10552555 · Cited by 3
“Thirdly, for essential modality interaction in multimodal sentiment analysis [17] [18] [19], we introduce an inter-modal complementary unit (IRMC) to extract shared semantic cues from other modalities through pairwise learning.”
MethodTextual Emotion Recognition: Advancements and Insights
Esmail, A. A., Abdallah, S. M., El-Sayed, M. A., Ghaith, A. E., Elbanhawy, M., Al‐Zoghby, A. M., & Moawad, I. F. (2024) · Novel Intelligent and Leading Emerging Sciences Conference (NILES) · doi:10.1109/niles63360.2024.10753241 · Cited by 3
“Self-supervised representation learning, as demonstrated by Yu et al. [34], incorporates a multi-task learning strategy for modality-specific representations, differences between modalities simultaneously, and learning consistency as shown in [35].”
GFIDF:gradual fusion intent detection framework
Yang, Q., Liu, Y., Lu, L., & Liu, L. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06708-3 · Cited by 4
“We use a series of state-of-the-art baselines, including text-based BERT methods, text classifiers, and a number of trimodal methods, including (1) MAG-BERT [49]; (2) Text-Classifier [50]; (2) MulT [39]; and (3) MISA [43].”
Efficient Transfer From Image-Based Large Multimodal Models to Video Tasks
Cao, S., Zhao, Z., Hao, S., Chai, W., Hwang, J., Wang, H., & Wang, G. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3557692 · Cited by 3
“Many methods [20], [25], [26] utilized pre-trained Convolu-tional Neural Network (CNN) [27] and Bidirectional Encoder Representations from Transformers (Bert) [28] to extract textual and video features, followed by the use of a Multi-Modal Transformer [29] for feature fusion.”
Multimodal intent recognition based on text-guided cross-modal attention
Li, Z., Peng, J., Lin, X., & Cai, Z. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06583-2 · Cited by 3
Multimodal representation fusion method for dense video captioning
Fang, H., Li, Y., & Li, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113856 · Cited by 3
Miao, X. (2025) · International Journal of High Speed Electronics and Systems · doi:10.1142/s0129156425406783 · Cited by 3
Meta-Learning for Incomplete Multimodal Sentiment Analysis
Tu, G., Wu, T., Luo, X., Zeng, X., Li, W., & Xu, R. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730231 · Cited by 4
Yu, J., Zhu, L., Chi, Y., Zhang, Y., Zheng, Y., Wang, Y., & Lu, X. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00572 · Cited by 3
“MISA [10] projects each modality into two distinct subspaces.”
Xie, W., Liang, H., Niu, Z., Hou, X., Song, S., Yu, Z., & Shen, L. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3767746 · Cited by 3
“We compare HFR-AME with the following SOTA models, i.e., TFN [56], LMF [27], MFM [42], Graph-MFN [59], MFN [57], MCTN [34], MulT [43], MISA [14], MAG-BERT [37], Self-MM [55], MMIM [13], FDMER [50], EMT-DLFR [41], MissModal [24], C-MIB [32], HyDiscGAN [47], and CASP [12] under the same dataset…”
Multimodal prototypical network for interpretable sentiment classification
Song, C., Chao, K., Jia, B., & Shen, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-19850-6 · Cited by 5
Multimodal Mixture of Low-Rank Experts for Sentiment Analysis and Emotion Recognition
Zhang, S., Zhang, J., Zhang, Z., & Li, L. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11210197 · Cited by 3
“2) Baselines: We compare the proposed method with various baseline models on both MSA and MER tasks.”
Bi-attention Modal Separation Network for Multimodal Video Fusion
Du, P., Gao, Y., & Li, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98358-1_46 · Cited by 3
Multimodal Sentiment Analysis using Prediction-based Word Embeddings
Kumar, T. P., & Vardhan, B. V. (2022) · International Conference on Edge Computing and Applications (ICECAA) · doi:10.1109/icecaa55415.2022.9936350 · Cited by 3
“Hazarika [2] proposed multisensory and modality-specific representations for mult imodal representation learning.”
BackgroundIMCN: Identifying Modal Contribution Network for Multimodal Sentiment Analysis
Zhang, Q., Shi, L., Liu, P., Zhu, Z., & Xu, L. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956114 · Cited by 3
“[8] proposed modality-invariant and modali-”
BackgroundTransformer-based Multimodal Contextual Co-encoding for Humour Detection
Deng, B., Tian, J., & Li, H. (2022) · International Conference on Culture-oriented Science & Technology (ICCST) · doi:10.1109/cost57098.2022.00067 · Cited by 3
Molecular Property Prediction based on Bimodal Supervised Contrastive Learning
Sun, Y., Islam, M., Zahedi, E., Kuenemann, M. A., Chouaib, H., & Hu, P. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995689 · Cited by 3
“Inspired by [9], we designed a network that maps each modality to two subspaces: joint embedding space and modality-specific space.”
BackgroundMethodMMSF: A Multimodal Sentiment-Fused Method to Recognize Video Speaking Style
Zhang, B., Fang, Y., Yu, F., Bei, J., & Ren, T. (2023) · International Conference on Multimedia Retrieval · doi:10.1145/3591106.3592219 · Cited by 3
“MISA [13] projects each modality to two distinct subspaces, and provides a holistic view of the multimodal data, which is used for fusion that leads to task predictions.”
BackgroundMultimodal Intent Recognition Based on Contrastive Learning
Wu, Y., Liu, Z., Su, Z., & Song, X. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-4402-6_13 · Cited by 3
Multimodal Prediction of User's Performance in High-Stress Dialogue Interactions
Gilani, S. N., Pollard, K. A., & Traum, D. R. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3610661.3617166 · Cited by 3
“This observation is consistent with the results reported in other studies[13, 18, 20, 32].”
ResultFeature Fusing with Vortex-Based Classification of Sentiment Analysis Using Multimodal Data
Kumar, V. S., Renukadevi, S., Yashaswini, B. M., Malagi, V. P., & Pareek, P. K. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-7622-5_32 · Cited by 3
Sentiment Analysis Using Bi-ConvLSTM
Matta, D. S., & Saruladha, K. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-99-6553-3_29 · Cited by 3
Feature Alignment and Reconstruction Constraints for Multimodal Sentiment Analysis
Zhu, Q., Lan, T., Liang, J. P., Xiang, D., & He, H. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650976 · Cited by 3
“MISA [19] and MMIM [20] attempt to preserve the unique semantics of unimodal features by feature space mapping and maximizing mutual information when learning multi-modal common features.”
BackgroundDual-Decoupling With Frequency-Spatial Domains for Image Manipulation Localization
Pan, W., Ma, W., Zhou, T., Zhao, S., Gu, L., Shi, G., & Xia, Z. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3472846 · Cited by 3
“The decoupling network has been widely used in many computer vision tasks [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], such as segmentation, recognition, and deep-fake detection.”
Kang, H. W., Choi, J. W., & Kim, B. H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78201-5_20 · Cited by 3
PSACF: Parallel-Serial Attention-Based Cross-Fusion for Multimodal Emotion Recognition
Zhang, Y., Liu, Y., & Cheng, C. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822849 · Cited by 4
“MISA [12] 85.50 52.20 85.30 PMR [13] 82.60 52.50 83.30 Self-MM [14] 82.81 - 82.53 SupArc [15] 84.57 55.01 84.71 UniSA [16] 85.86 54.39 - TriDiRA [17] 84 Our method outperforms all baseline models in ACC 2 and F1 Score on both datasets, showing competitive performance.”
InfluentialMulti-Noise Representation Learning for Robust Speaker Recognition
Cho, S., & Wee, K. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3530879 · Cited by 3
“Multi-modal learning addresses heterogeneous information from multiple modalities and has gained increasing attention in various fields, including video/audio classification and retrieval [16], [17], segmentation [18], [19], and emotion recognition [20], [21].”
Modality-Guided Refinement Learning for Multimodal Emotion Recognition
Cho, S. (2025) · IEEE Access · doi:10.1109/access.2025.3554708 · Cited by 4
“Secondly, compared to feature-decoupling-based methods [14], [15], [17], [19], our method shows consistent improvements, demonstrating the effectiveness of modality-guided losses in enhancing decoupled modality representations.”
InfluentialCross Domain Features for Subject-Independent Stress Detection
Radhika, K. R., & Oruganti, V. R. M. (2022) · IEEE Region 10 Symposium (TENSYMP) · doi:10.1109/tensymp54529.2022.9864379 · Cited by 3
“Motivated by the success of MI learning for different applications [10] [11], [12], [13], we”
MethodDistribution Distance Regularized Sequence Representation for Text Matching in Asymmetrical Domains
Yu, W., Xu, C., Xu, J., Pang, L., & Wen, J.-R. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3145289 · Cited by 3
“Existing studies [12]–[15] have shown that it is critical for the projection network to generate modal-invariant features.”
BackgroundNRAFN: a non-text reinforcement and adaptive fusion network for multimodal sentiment analysis
Wei, J., & Shao, X.-H. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19433-z · Cited by 2
Modality-specific and -shared Contrastive Learning for Sentiment Analysis
Liu, D., You, J., Xie, G., Lee, L., Wang, F. L., & Yang, Z. (2024) · International Conference on Multimedia Retrieval · doi:10.1145/3652583.3658004 · Cited by 2
Zhang, Y., Zhong, H., Chen, G., Alhusaini, N., Zhao, S., & Wu, C. (2024) · International Conference on Networking and Network Applications (NaNA) · doi:10.1109/nana63151.2024.00088 · Cited by 3
“To address this issue, a recent work [9] proposed a flexible multimodal learning framework that utilizes invariant mapping to capture latent commonalities and related features before fusion.”
BackgroundMethodInfluential
Metadata
- Authors
- Hazarika, D., Zimmermann, R., & Poria, S.
- Affiliations
- National University of Singapore (SG)
- Singapore University of Technology and Design (SG)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 1122-1131
- OpenAlex
- W3093051361
- Cited by
- 1,073
- References
- 45
- Citations by year
- 2026: 280 · 2025: 315 · 2024: 238 · 2023: 136 · 2022: 78 · 2021: 23 · 2020: 3
- FWCI
- 22.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- Funders
- Ministry of Education - Singapore (MOE2018-T2-1-103)
- DSO National Laboratories - Singapore (RTDST190702)
- RIE 2020 Advanced Manufacturing and Engineering (AME) programmatic grant (A19E2b0098)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, modality gap, modality-invariant representations, modality-specific representations, subspace projection, multimodal fusion, user-generated video, humor detection, MOSI, MOSEI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
