The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Open accessAbstract
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On CMU-MOSEI dataset, our proposed C-MIB yields 0.7% improvement on Acc2 and 1.2% improvement on F1 score compared to the current state-of-the-art MISA [53]. The performance on Acc7 and Corr is also improved compared to the state-of-the-art methods.”
From Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations (2023) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 8×
“Inter-utterance context. et al. proposed one of the first models, bc-LSTM , which utilized”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“However, following recent works [7], including the stateof-the-art ICCN [50], we utilize the pre-trained BERT [11] as the feature extractor for textual utterances.”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“ndard low-level features that are provided by the respective benchmarks and utilized by the state-of-the-art methods. 4.3.1 Language Features. Traditionally, language modality features has been GloVe [38] embeddings for each token in the utterance. However, following recent works [7], including the stateof-the-art ICC…”
MethodInfluential(2008) · Journal of Machine Learning Research · Cited by 33,051
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 5×
“licly available models of [26, 51, 58]).”
BackgroundInfluentialSelf-citationEkman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,971 · cited 1×
“Both MOSI and MOSEI use Facet 3 to extract facial expression features, which include facial action units and face pose based on the Facial Action Coding System (FACS) [14].”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · cited 1×
“For UR_FUNNY, OpenFace [4], a facial behavioral analysis tool, is used to extract features related to the facial expressions of the speaker.”
Method
Research works printed but not linked: 15
- Andrew Galen, 2013, Proceedings of the 30th International Conference on Machine Learning, ICML 2013 (JMLR Workshop and Conference Proceedings, 1255
- Bousmalis Konstantinos, Advances in Neural Information Processing Systems 29. Curran Associates
- 10.5555/2354409.2355082 · DOI
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 19
- Drugman Thomas, 2011, Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonics. In INTERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association. ISCA
- Kiela Douwe, 2019, NeurIPS 2019 Workshop. Curran Associates, Inc.
- Lu Jiasen, Advances in Neural Information Processing Systems 32. Curran Associates
- 10.5555/2392963.2392965 · DOI
- Peng Minlong, 2018, Long Papers
- Rajagopalan Shyam Sundar, 2016, Proceedings, Part VII (Lecture Notes in Computer Science, 353
- Salzmann Mathieu, 2010, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010 (JMLR Proceedings, 708
- Hubert Tsai Yao-Hung, 2019, Learning Factorized Multimodal Representations. In 7th International Conference on Learning Representations, ICLR
- Vaswani Ashish, Advances in Neural Information Processing Systems 30. Curran Associates
- Zadeh Amir, 2018, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, 2246
- Zellinger Werner, 2017, Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning. In 5th International Conference on Learning Representations, ICLR
Cited by
Sorted by citations
Items 801–900 of 1,001
Wang, C., Liang, J., Cao, P., Yang, J., & Zai͏̈ane, O. R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5766415
PMAAN: a priority-guided multi-scale adaptive attention network for multimodal sentiment analysis
Xu, F., An, S., Guo, D., & Zhang, X. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-08071-3
DMBL: Dual-Stage Multimodal Balanced Learning for Multimodal Sentiment Analysis
Tang, Y.-C., Chen, Q., Zheng, N., & Luo, N. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4367-0_6
MRF: A Modality-Resilient Framework for Handling Missing Modalities in Multimodal Learning
Du, P., Huang, Y., Chang, X., & Li, R. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3346-6_2
CoMPLe: Cross-Modal Hybrid Prompt Learning for End-to-End Multimodal Emotion Recognition
Feng, J., Zhao, Z., Zhou, L., Pu, Y., Jieping, Y. E., Xu, D., & Gu, J. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3638415
“Research has shown that in MER tasks, information from the text modality is stronger than the other two modalities [13], [39].”
Multilevel Feature Fusion and Attention Enhancement for Multimodal Sentiment Analysis
Zhang, L., He, T., Chen, B., & Shuan, C. (2025) · International Symposium on Computer and Information Processing Technology (ISCIPT) · doi:10.1109/iscipt67144.2025.11265256
“Based on previous studies [22]–[24], this paper uses the BERT [25] model for text modality processing.”
Mian, D. (2025) · IEEE International Conference on Sensors, Electronics and Computer Engineering (ICSECE) · doi:10.1109/icsece65727.2025.11256961
“Later, Devamanyu et al [16] proposed a novel framework structure that aims to project each modality into two different subspaces.”
MDR-MSA: multi-perspective decoupled representation learning for multimodal sentiment analysis
Du, J., Zhi, Y., Wang, H., Wang, K., & Li, J. (2025) · Multimedia Systems · doi:10.1007/s00530-025-02062-3 · Cited by 1
Zheng, Y., Zhang, C., Wen, Y., Qi, X., & Zhang, P. (2025) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3743093.3770990
CDFN: Credibility-Driven Fusion Network for Multimodal Sentiment Analysis
Liu, C., Liu, C., & Wei, X. (2025) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3743093.3771040
“…of CDFN, we compare it with representative state-of-the-art methods, including TFN [24], MFN [25], MuIT [16], ICCN [14], MAG [13], MISA [5], TFR-Net [23], MHE [4], CMHFM [19], and TMBL [7], covering diverse fusion paradigms such as tensor composition, attention, alignment, and graph…”
Dynamic Text-guided Multi-task Transformer for Multi-modal Sentiment Analysis
Huang, W. R., Cai, W., & She, Z. (2025) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3743093.3770944
Chen, Z., Zhong, H., Alhusaini, N., & Chen, G. (2025) · International Conference on Networking and Network Applications (NaNA) · doi:10.1109/nana66698.2025.00066
“In the unimodal sentiment analysis, [13] endeavored to disentangle modality features within a unified space to present both modality-invariant and modality-specific representations.”
InfluentialShen, Q., Xin, J., Dai, B. T., Huang, Z., & Wang, Z. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3639945
“Several utterance-level or global feature-level fusion methods have achieved encouraging results using attention mechanisms [28], [29] or tensor-based fusion [30], [31], [32], but they have failed to achieve fine-grained multimodal information mining.”
GCA: gated network-integrated cross-attention for multimodal sentiment analysis
Meng, C., Zhang, Y., & Gu, J. (2025) · doi:10.1117/12.3092518
“…time series from speech and image data, then dynamically converts the extracted non-linguistic context features and text features; Hazarika et al. [5] proposed a new framework containing two different network structures, where one network learns the common representation features between…”
Multimodal dynamic collaborative sentiment analysis method
Guan, S., Chen, L., Huang, X., Liu, Y., Du, Y., & Wang, W. (2025) · International Conference on Artificial Intelligence and Pattern Recognition · doi:10.1117/12.3093456
Liu, X., Wang, T., Cui, A.-X., & Zhang, X. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-02198-9
Liu, X., Ma, Z., Li, M., Pu, T., & Wang, G. (2025) · International Conference on Computer Vision and Data Mining (ICCVDM) · doi:10.1109/iccvdm66874.2025.11290084
Wang, M., & Zhong, W. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.11.036
Liu, W., Zhang, C., Shun-shi, H., & Zhang, J. (2025) · International Conference on Intelligent Computing and Human-Computer Interaction (ICHCI) · doi:10.1109/ichci67614.2025.11292502
“For example, Haz-arika et al. introduced the MISA model, which projects features into modality-invariant and modality-specific subspaces, and designs similarity/difference loss functions to enhance discriminability [7].”
InfluentialMDAN: Multimodal Dynamic Awareness Network for Multimodal Aspect-Based Sentiment AnalysisPreprint
Zhang, H., Xu, W., Peng, J., Zhang, X., & Zhang, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5490608
Wang, J., & Jiang, Y. (2025) · International Conference on Information Science, Computer Technology and Transportation (ISCTT) · doi:10.1109/isctt66403.2025.11137837
A Multimodal Fusion Adapter for Enhanced Chinese Sentiment Analysis
Peng, S.-W., Ho, T.-Y., & 蘇順豐 (2025) · IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan) · doi:10.1109/icce-taiwan66881.2025.11208071
DRLHomo: Disentangled Representation Learning for Cross-Modal Homography Estimation
Li, T., Zhou, Z., Zhu, Q., Luo, J., & Wang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5702-8_1
Wang, Y., Liu, Y., Huang, X., Huang, Y., & Wang, Y. (2025) · International Symposium on IoT and Intelligent Robotics (IoTIR) · doi:10.1109/iotir66925.2025.00023
A UMRFormer-enhanced ALMT with modality-specific attentions for multimodal sentiment analysis
Ding, R., & Wang, K. (2026) · Signal Image and Video Processing · doi:10.1007/s11760-025-05067-7
Integrated feature-enhanced multimodal intent detection method
Yang, Q., Lu, L., Liu, Y., & Song, J. (2026) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02876-7
Research on Model Fusion and Enhancement Methods for Multimodal Emotional Understanding
Li, Z. (2025) · International Conference on Cloud Computing, Big Data Application and Software Engineering (CBASE) · doi:10.1109/cbase67452.2025.11335566
“MISA [18] retains both cross-modal and modality-unique features, achieving strong robustness and accuracy.”
Chan, S., Zhai, Y., Li, Y., Wang, Y., & Mao, J. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3656117
“[41]inMISAtodecouplemodality-invariantand modality-specificfeatures,thoughMISAremainedtext-reliant.”
Lightweight and Interpretable Multimodal Sentiment Analysis via Triple Text Enhancement
Han, J., Ma, Y., & Hu, Y. (2025) · International Conference on Cloud Computing, Big Data Application and Software Engineering (CBASE) · doi:10.1109/cbase67452.2025.11335501
Multimodal Sentiment Analysis via Spatio-Temporal Decoupling and Language-Focused Fusion
Li, J., Zhang, L., Cui, H., Wang, J., & Li, R. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5567-3_17
Zhou, H., Liu, J., Li, X., Liu, Y., & He, H. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3658336 · Cited by 2
“MSA is broadly defined as the exploration of subjective factors, such as an individual’s specific emotional attitudes towards other people and things [29], [30], [31], [32], [33].”
Multimodal Sentiment Analysis Based on Incomplete Data
Yang, C., Liu, J., Sun, Y., Yang, Y., & Zhang, J. (2025) · International Symposium on AI and Cybersecurity (ISAICS) · doi:10.1109/isaics66888.2025.11350257
Video-guided global contrast learning for multimodal sentiment analysis
Zhang, E., Zhao, Y., Qiao, X., & Li, L. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11342476
Sentiment Intensity Contrastive Text‐Enhanced Fusion Network
Jiang, H., Shi, L., Kong, D., Hua, J., Shang, H., & Chen, L. (2026) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70593
“Following prior work [14, 18, 31, 38], for the MOSI and MOSEI datasets the regression task is evaluated using mean absolute error (MAE) and the Pearson correlation coefficient (Corr); the classification task is evaluated using 7-way accuracy (Acc-7), binary accuracy (Acc-2) and the F1 score.”
Multi-Scale Fusion with Explicit Word-Level Alignment for Multimodal Sentiment Analysis
Li, X., Xing, J., Ma, Y., An, X., & Ren, Y. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343683
GSAW: A Guided Synergistic Adaptive Weighting Framework for Fine-Grained Sentiment Analysis
Liu, Z., Zhang, J., Shi, Y., Li, Z., Yang, X., Jiao, Z., & Zhang, X. (2025) · IEEE International Conference on Advanced Infocomm Technology · doi:10.1109/icait66450.2025.11353338
“This direction was prominently established by frameworks like MISA, which projects modalities into separate invariant and specific subspaces[2].”
Multimodal Sentiment Analysis Based on Uncertain Missing Patterns
Xu, B., & Yang, J. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11342525
“For medium-term fusion[5][8][39], Multimodal Transformer (MulT)[5]employed a cross-modal attention mechanism, MISA[18] proposed a dual-stream architecture to mitigate modal heterogeneity, and Yang et al.'s[8] hierarchical Transformer framework enhanced feature fusion completeness through context-aware modules and cros…”
Zhao, X., Yang, S., Tang, B., & Jiang, R. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3660288
“Multimodal data often exhibits gaps in feature space distributions [33], [53].”
Xu, Y., & Huan, H. (2026) · Multimedia Systems · doi:10.1007/s00530-025-02138-0
“Hazarika et al. [ 6] applied disentangled rep - resentation learning to multimodal sentiment analysis and introduced the MISA model, which improves model inter - pretability by decomposing modality-specific features into modality-invariant and modality-dependent components.”
SADGR: Adaptive Cross-Modal Emotion Recognition via Self-Supervised Alignment and Dynamic Gating
Zhang, J., Mao, J., Hou, Y., Noh, G., & Zhang, F. (2026) · IEEE Access · doi:10.1109/access.2026.3663574 · Cited by 1
“Models like MulT (Multimodal Transformer) [10] treat inter-modal interactions as a ‘translation’ process, guiding information flow through paired cross-modal Transformers.”
InfluentialVisual-based Sentiment Analysis Chatbot
Bhardwaj, H., Rajesh, H., Reddy, J. R. D., Kishore, G., Neelima, N., & Glace Varghese, T. (2025) · International Conference on Ubiquitous Computing and Intelligent Information Systems (ICUIS) · doi:10.1109/icuis67429.2025.11380658
IDBF: Information-theoretic Disentanglement with Bridge Fusion for Multimodal Sentiment Analysis
Qi, W. (2025) · International Conference on Embodied Intelligence and Large Models · doi:10.1145/3788731.3788748
Multi-level perception cross-modal fusion framework for multimodal sentiment analysis
Yu, L., & Liu, A. (2026) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-026-02991-z
“2, where N layer stands for multi-layer processing publications: TFN [9], LMF [10], MuLT [11 ], MISA [ 4], Self-MM [6], CubeMLP [34] and TMBL [35], as well as text-centric methods: MMIM [ 7], BBFN [36 ], TETFN [37] and ALMT [13], MMuReLAF [38] and ULMD [39], KuDA [40] and MAG+ [41 ], SKEAFN [ 42]…”
Deng, L., Wang, R., Lu, Z., Zhu, Y., & Li, Z. (2025) · International Conference on Computing, Communication, Perception and Quantum Technology (CCPQT) · doi:10.1109/ccpqt66408.2025.11383052
Gavali, P. R., Patil, N., Ambi, S., Patil, P., Bandivadekar, M., & Salokhe, A. (2025) · IEEE International Conference on Blockchain and Distributed Systems Security (ICBDS) · doi:10.1109/icbds67396.2025.11376704
AG-MSA: Adaptive Gated Prompt Learning for Few-Shot Multimodal Sentiment Analysis
Zhang, A., Guo, X., Zhu, T., & Yu, H. (2026) · International Conference on Consumer Electronics and Computer Engineering (ICCECE) · doi:10.1109/iccece69169.2026.11399847
“MISA [10]: Project different modes into invariant subspace and specific subspace to learn decoupled modal representation.”
RMSAGF: A synergistic generation and fusion framework for robust multimodal sentiment analysis
Li, Y., Gao, B., Li, X., Li, L., Li, Y., Liu, S., & Liu, Z. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131841
Wang, Y., Xie, W., Yang, Z., Wang, L., Liu, M., Chen, Y., & Gao, Y. (2026) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2026.3668198
“For example, modality-invariant and -specific representations (MISA) [59] utilizes a composite of losses comprising distributional similarity and orthogonality constraints to disentangle each modality’s features into shared and unique subspaces.”
DMF:A plugin based on dynamic multi-scale fusion for multimodal sentiment analysis
Wu, J. Y., & Chen, W. (2025) · International Conference on Artificial Intelligence, Human-Computer Interaction and Robotics (AIHCIR) · doi:10.1109/aihcir67580.2025.11405002
“We compare the performance of our proposed model with previous state-of-the-art approaches, including TFN [22], LMF [23], MISA [24], MAG-BERT [10], Self-MM [16], MMIM [25], CHFN [12], UniMSE [5], the BART-, T5, and GPT2-based variants of UniSA [26], AV-MC, MMGCN [27], MM-DFN [28], EmoCaps [29],…”
Advancing Multimodal Sentiment Analysis: Models, Datasets, and Future Directions
Ahmad, S., Agarwal, P., Alankar, B., & Naaz, S. (2025) · IEEE DELCON - International Conference on Recent Smart Technologies in Engineering for Sustainable Development (DELCON) · doi:10.1109/delcon68055.2025.11400288
“New models like MISA [12] improve performance by separating common and unique information from each modality.”
DGGD: A Unified and Discriminative Representation Learning Method for Multimodal Sentiment Analysis
Ye, L., Zheng, N., Li, Y., & Luo, N. (2025) · International Conference on Computer, Internet of Things and Smart City (CIoTSC) · doi:10.1109/ciotsc67482.2025.11412996
“Following previous works [1], [9], [20], [2], we report several classification metrics: 7-class accuracy (Acc-7), 5-class accuracy (Acc-5), binary accuracy (Acc-2) and F1-score(F1).”
InfluentialEmo-CDFN: Emotion Lexicon Enhanced and VA-Aware Network for Multimodal Emotion Recognition
Zhang, S., He, B., Xu, T., Liang, Y., & Peng, B. (2025) · International Conference on Computer, Internet of Things and Smart City (CIoTSC) · doi:10.1109/ciotsc67482.2025.11413135
“Hazarika et al. [4] proposed the MISA model, which employs adversarial learning to decouple modality-invariant and modality-specific representations from multimodal signals.”
Research on Multi-Modal Big Data Fusion Classification Algorithm Based on Attention Mechanism
Zhao, Q., Zhang, L., Qin, J., Yang, M., & Chen, H. (2025) · International Conference on Big Data Applications, Mechatronics Engineering and Automation (BDAMEA) · doi:10.1109/bdamea68159.2025.11406724
“There are distribution gaps between diverse sources of data such as different modality sources that can be difficult for fusion techniques to overcome[3].”
A Multimodal Semantic Adversarial Alignment Approach for Sentiment Analysis
Wei, J., Wang, J., Jin, X., & Wang, G. (2025) · International Conference on Virtual Reality and Visualization · doi:10.1109/icvrv67992.2025.00149
“MISA [11] and MInd [12] adopt shared-private representation learning frameworks to disentangle shared and private modality features, yet lack mechanisms for fine-grained cross-modal interaction.”
Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities
Bao, J., Qian, J., Yan, M., & Yang, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37098-3_18
Exploiting Body Gesture and Automated Speech Recognition for Multimodal Sentiment Analysis
Zeng, J., Nie, T., Chen, W., Huang, Q., & Huang, Q. Y. (2025) · International Conference on the Frontiers of Electronic, Electrical and Information Engineering (ICFEEIE) · doi:10.1109/icfeeie66944.2025.00035
“Most existing works employed the two-phase pipeline to extract modal representations [2], [3], [12].”
Lu, L., Yang, Q., & Liu, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7075-1_26
Wang, A., Li, M., Zeng, N., & Cao, J. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2639662
Efficient Fusion for Multimodal Sentiment Analysis: The DSOAN Approach
Tian, R., Kang, X., Matsumoto, K., & Yoshida, M. (2025) · IEEE International Conference on Artificial Intelligence, Computer Science and Information Processing (AICSIP) · doi:10.1109/aicsip65423.2025.11427290
“It achieves these results with only 2.0 million parameters, significantly lower than MISA [24] (4.3 M), Self-MM [25] (4.7 M), and MM-ALIGN [26] (6.1 M).”
InfluentialChen, R., Zhang, X., Cheng, S., Zhang, W., Zhang, H., & Zhang, W. (2026) · Applied Sciences · doi:10.3390/app16062952
C2MF: Confidence-Calibrated Multimodal Fusion for Continuous Sentiment Regression
Liu, J., Qi, K., Liao, Z., Yuan, F., Zhuo, W., & Zhou, Y. (2026) · IEEE Signal Processing Letters · doi:10.1109/lsp.2026.3677608
“Another effective direction is to disentangle shared semantics from modality-specific factors, which helps reduce redundancy and distribution mismatch across modalities [19], [20], [21].”
Bingqian, Q., Lu, W., Yushuo, Z., & Xuyang, W. (2026) · IEEE International Conference on Power, Electronics and Green Energy (ICPEGE) · doi:10.1109/icpege67691.2026.11451575
Chen, Z., & Tian, J. (2026) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70657
LGBE-MSA: Language-Guided Bidirectional Encoder for Multimodal Sentiment AnalysisPreprint
Zhang, W., Gao, B., Chen, Y., Fu, Q., Li, Y., Liu, S., & Liu, Z. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6486772
Meme-Based Sentiment Shift Prediction in Financial Markets
Edirisingha, E. D. L. L. S., & Samarasinghe, T. D. (2026) · International Conference on Advanced Research in Computing (ICARC) · doi:10.1109/icarc68737.2026.11454013
“Fusion methods have evolved from simple concatenation to attention-based cross-modal learning [21], [22].”
SCoPE: Shift-Aware Speaker-Conditioned Priors for Emotion Recognition in ConversationsPreprint
Kaplan, B. C., & Wermter, S. (2026) · Research Square · doi:10.21203/rs.3.rs-9065619/v1
Neural methods for spoken dialogue understanding
Chapuis, É. (2021) · theses.fr (ABES) · doi:10.70675/1fc9d449zab9az41c9zaa43z5f3ca7ec8ba4
Xu, Y., Chen, H., & Tan, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6551274
CEAT: Context-Emotion Adversarial Training Framework for Robust Emotion-Driven Fraud Detection
Li, C., Wu, S., Lu, Y., Ma, Y., Liu, J., Jia, D., Han, M., Li, F., & Hu, P. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792981
Sentiment analysis of social media for enhancing disaster response strategies
Chen, Y., Hu, W., & Zeng, L. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1658777
Sentiment Analysis Based on Super Learning Multimodal Transformer
An, D., & Wang, S. (2026) · IEEE Signal Processing Letters · doi:10.1109/lsp.2026.3682989
文敬, 野., Xiang, L., Zhao, X., & Zong, C. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3683357
“Another typical framework is MISA [27], which projects each modality into a modality-specific subspace to capture its unique features and into a modality-invariant sub-space to capture its shared characteristics.”
Zhang, J., Deng, H., Li, Q., Li, D., & Yang, T. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3681783
“For the CMU-MOSEI dataset, comparisons are made with MISA [30], MAG-BERT [31], MMIN [32], Self-MM [33], CubeMLP [27], DiCMoR [34], GCNet [35], HyCon [36], MSG-MBA [37], EMT [38], JOSFD [4], and MMML [5].”
Prompt-Enhanced Multimodal Learning for Robust Sentiment Analysis with Incomplete Data
Sun, Y., Zhang, P., Zhao, W., Wang, F., Liu, X., & Wu, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7078-2_4
Zhang, Y., Chen, Q., Huang, Y., Xu, H., Qi, B., & Zhou, L. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7078-2_1
Selective Hub Fusion with Modality-Heterogeneous Experts for Multimodal Emotion Recognition
Zhao, H., Xiong, L., & Wang, K. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463655
LETPAV: Lexicon-Enhanced Text with Progressive Audio-Visual Fusion for Multimodal Sentiment Analysis
Li, J., Wang, Y., Li, W., Chen, J., & Wang, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461822
Liu, L., Zhang, Y., Wang, J., Xu, D., & Zhang, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464008
DDSR-Net: Robust Multimodal Sentiment Analysis via Dynamic Modality Reliability Assessment
Hou, J., Zuo, E., Shi, C., & Ubul, K. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462473
A-MSA: Adversarial Feature Disentanglement for Multimodal Sentiment Analysis
Zhou, L., Feng, J., Xiong, H., Pu, Y., & Zhao, Z. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462173
Meng, C., Zhou, Z., He, L., Du, X., Ouyang, C., & Gan, Z. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463049 · Cited by 3
Geng, R., Sun, Q., Cao, H., & Wang, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460389
Multimodal Variational Graph Network for Multimodal Sentiment Analysis
Ren, Y., Lu, Q., Long, Y., Zhu, Z., Meng, J., & Pei, H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11465145
CaMoD: Causal-Aware Modality Denoising for Multimodal Dialogue Intent Recognition
Zhang, J., Li, B., & Liu, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462418
GRNet: Graph Reconstruction Network for Robust Multimodal Sentiment Analysis
Xu, Z., Tian, L., Zhang, P., Peng, X., & Yao, H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463932
Token-level refined region-based framework for multimodal sentiment analysis
Li, Y., He, Q., Lv, M., Lin, Q., & Jiang, T. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113803
Ye, M., Yin, M., & Guo, Y. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3687649
“(cid:2) MISA [11]: Projects multimodal features into shared and specific subspaces, then effectively fuse them.”
Text-enhanced cross-modal reinforced fusion network for multimodal sentiment analysis
Zhao, W., & Wu, Y. (2026) · Pattern Analysis and Applications · doi:10.1007/s10044-026-01676-1
DHFN: a disentangled hierarchical fusion network for multimodal sentiment analysis
Yan, J., & Shen, J. (2026) · Pattern Analysis and Applications · doi:10.1007/s10044-026-01671-6
“Hazarika et al. [ 11] proposed MISA, which decomposes each modality’s features into modality-invariant and modality-specific features.”
CMEF-Net: A Hierarchical Cross-Modality Fusion Framework for Deep Multimodal Emotion Recognition
Narayana, M. L., Reddy, B. G. O., & Reddy, T. R. (2026) · International Conference on Emerging Smart Computing and Informatics (ESCI) · doi:10.1109/esci68015.2026.11493418
Register-augmented attention and self-calibrated fusion for robust multimodal sentiment analysis
Xu, B., Wang, W., & Ding, A. (2026) · Multimedia Systems · doi:10.1007/s00530-026-02653-8
Miyoshi, R., Otani, M., & Okafuji, Y. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00476
“Models trained on complete inputs, such as MulT [48], MISA [13], and DMD [22], tend to perform well when evaluated under full modality conditions.”
Shuai, Q., Chen, X., Hu, F., & Cheng, Y. (2026) · Complex & Intelligent Systems · doi:10.1007/s40747-026-02318-z
“We compared FMCL with current state-of-the-art methods using the same dataset settings, including EF-LSTM, LF-LSTM, TFN [17], LMF [16], MFM [18], RAVEN [47], MCTN [48], MulT [27], MISA [49], DMD [19], Self-MI [50], CMHFM [51], TCAN [52], DLF [53], Coupled Mamba [36], DER-GCN [54].”
Dual-Domain Multimodal Hyperbolic Fusion for Cardiopulmonary Disease Diagnosis in Emergency Care
Nan, K., Samaan, M., Burns, B., Ning, X., Han, Y., & Xue, Y. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00839
“….858 ± 0.005 0 .768 ± 0.002 0 .726 ± 0.003 DAFT [46] 0.790 ± 0.004 0 .786 ± 0.004 0 .627 ± 0.009 0 .857 ± 0.005 0 .774 ± 0.003 0 .726 ± 0.002 MISA [17] 0.785 ± 0.003 0 .782 ± 0.003 0 .615 ± 0.011 0 .876 ± 0.001 0 .770 ± 0.006 0 .700 ± 0.002 DDMF-Net (Ours) ∗ 0.818 ± 0.002 0 .812 ± 0.003 0 .669 ±…”
InfluentialA Hybrid Sensor-Fusion Framework for Linguistically Complete Sign Language Recognition
Warnasooriya, I. S. B., Sudantha, B. H., & Premachandra, C. (2026) · International Conference on Image Processing and Robotics (ICIPRoB) · doi:10.1109/iciprob69625.2026.11497804
“• The identification of four concrete research gaps in multimodal SLR, specifically addressing challenges in temporal asynchrony and sensor uncertainty [13], [14], [15], [16], [17].”
MEERA: Multimodal experts for evidence routing and sparse LLM adaptation in sentiment analysis
Niu, F., Liao, Y., Peng, X., Huang, H., & Zhang, B. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113922
TDGN: A text-guided dual-gated network for multimodal sentiment analysis
Xiao, W., Zhang, L., & Xu, Y. (2026) · PLoS ONE · doi:10.1371/journal.pone.0349024
Vignesh, V., J, S. K., Senthilkumar, A., & Jeevitha, R. (2026) · International Conference on Inventive Computation Technologies (ICICT) · doi:10.1109/icict68280.2026.11510716
“Due to common issues with facial emotion recognition, such as intra-class variability and overlapping of subtle expressions, the performance in other classes is lower [2], [6].”
InfluentialA Comparative Study of Multimodal Social Media Sentiment Analysis Across Different Fusion Strategies
Li, S., Zhang, Y., Pan, W., Xiong, L., & Li, C. (2026) · International Conference on Artificial Intelligence and Digital Services · doi:10.1145/3803686.3803688
Qi, W., & Peng, H. (2026) · International Conference on Computer, Artificial Intelligence and Control Engineering (CAICE) · doi:10.1145/3804601.3804624
Metadata
- Authors
- Hazarika, D., Zimmermann, R., & Poria, S.
- Affiliations
- National University of Singapore (SG)
- Singapore University of Technology and Design (SG)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 1122-1131
- OpenAlex
- W3093051361
- Cited by
- 1,073
- References
- 45
- Citations by year
- 2026: 280 · 2025: 315 · 2024: 238 · 2023: 136 · 2022: 78 · 2021: 23 · 2020: 3
- FWCI
- 22.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- Funders
- Ministry of Education - Singapore (MOE2018-T2-1-103)
- DSO National Laboratories - Singapore (RTDST190702)
- RIE 2020 Advanced Manufacturing and Engineering (AME) programmatic grant (A19E2b0098)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, modality gap, modality-invariant representations, modality-specific representations, subspace projection, multimodal fusion, user-generated video, humor detection, MOSI, MOSEI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
