Bimodal fusion network with multi-head attention for multimodal sentiment analysis
Open accessAbstract
The enrichment of social media expression makes multimodal sentiment analysis a research hotspot. However, modality heterogeneity brings great difficulties to effective cross-modal fusion, especially the modality alignment problem and the uncontrolled vector offset during fusion. In this paper, we propose a bimodal multi-head attention network (BMAN) based on text and audio, which adaptively captures the intramodal utterance features and complex intermodal alignment relationships. Specifically, we first set two independent unimodal encoders to extract the semantic features within each modality. Considering that different modalities deserve different weights, we further built a joint decoder to fuse the audio information into the text representation, based on learnable weights to avoid an unreasonable vector offset. The obtained cross-modal representation is used to improve the sentiment prediction performance. Experiments on both the aligned and unaligned CMU-MOSEI datasets show that our model achieves better performance than multiple baselines, and it has outstanding advantages in solving the problem of cross-modal alignment.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Scale differences and noise in large-scale unlabeled data can affect alignment accuracy.
“Other challenges include scale differences and noise in large-scale unlabeled data, which can affect alignment accuracy [178].”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [178]
The Introduction discusses misalignment between text and audio, and the Results discuss aligning their representations, but neither links alignment accuracy to scale differences or noise in large-scale unlabeled data. The Dataset section describes manually annotated data. Searches for scale differences, noise, and unlabeled data found no passage addressing the claimed causes. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370 · see citing works
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259 · see citing works
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · see citing works
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · see citing works
Memory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855 · see citing works
Mao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568 · see citing works
Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional networks
Liang, B., Su, H., Gui, L., Cambria, E., & Xu, R. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107643 · Cited by 540 · see citing works
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Wang, Y., Shen, Y., Liu, Z., Liang, P. P., Zadeh, A., & Morency, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017216 · Cited by 493 · see citing works
Sun, Z., Sarma, P., Sethares, W., & Liang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6431 · Cited by 410 · see citing works
Learning Factorized Multimodal Representations
Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., & Salakhutdinov, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06176 · Cited by 323 · see citing works
Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
Han, W., Chen, H., Gelbukh, A. F., Zadeh, A., Morency, L., & Poria, S. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479919 · Cited by 261 · see citing works
Class-level spectral features for emotion recognition
Bitouk, D., Verma, R., & Nenkova, A. (2010) · Speech Communication · doi:10.1016/j.specom.2010.02.010 · Cited by 219 · see citing works
Learning Alignment for Multimodal Emotion Recognition from Speech
Xu, H., Zhang, H., Han, K., Wang, Y., Peng, Y., & Li, X. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3247 · Cited by 185 · see citing works
Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment
Gu, Y., Yang, K., Fu, S., Chen, S., Li, X., & Marsic, I. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1207 · Cited by 160 · see citing works
VistaNet: Visual Aspect Attention Network for Multimodal Sentiment Analysis
Truong, Q.-T., & Lauw, H. W. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301305 · Cited by 154 · see citing works
Speech Emotion Recognition Using Multi-hop Attention Mechanism
Yoon, S., Byun, S., Dey, S., & Jung, K. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683483 · Cited by 138 · see citing works
Recognizing Emotions in Video Using Multimodal DNN Feature Fusion
Williams, J., Kleinegesse, S., Comanescu, R., & Radu, O. (2018) · Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML) · doi:10.18653/v1/w18-3302 · Cited by 132 · see citing works
Yang, K., Xu, H., & Gao, K. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413690 · Cited by 132 · see citing works
Wu, Y., Lin, Z., Zhao, Y., Qin, B., & Zhu, L.-N. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.417 · Cited by 129 · see citing works
Memory Fusion Network for Multi-view Sequential LearningPreprint
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00927 · Cited by 117 · see citing works
Context- and Sentiment-Aware Networks for Emotion Recognition in Conversation
Tu, G., Wen, J., Liu, C., Jiang, D., & Cambria, E. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3149234 · Cited by 86 · see citing works
Multi-Fusion Residual Memory Network for Multimodal Human Sentiment Comprehension
Mai, S., Hu, H., Jia, X., & Xing, S. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3000510 · Cited by 66 · see citing works
Fusion Techniques for Utterance-Level Emotion Recognition Combining Speech and Transcripts
Sebastian, J., & Pierucci, P. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3201 · Cited by 63 · see citing works
Audio‐Textual Emotion Recognition Based on Improved Neural Networks
Cai, L., Hu, Y., Dong, J., & Sitong, Z. (2019) · Mathematical Problems in Engineering · doi:10.1155/2019/2593036 · Cited by 62 · see citing works
Fusion Approaches for Emotion Recognition from Speech Using Acoustic and Text-Based Features
Pepino, L., Riera, P., Ferrer, L., & Gravano, A. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054709 · Cited by 61 · see citing works
Speech Emotion Recognition Based on Speech Segment Using LSTM with Attention Model
Atmaja, B. T., & Akagi, M. (2019) · IEEE International Conference on Signals and Systems (ICSigSys) · doi:10.1109/icsigsys.2019.8811080 · Cited by 57 · see citing works
Xiao, L., Wu, X., Wu, W., Yang, J., & He, L. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747542 · Cited by 49 · see citing works
Multimodal Speech Emotion Recognition Using Cross Attention with Aligned Audio and Text
Lee, Y., Yoon, S., & Jung, K. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2312 · Cited by 30 · see citing works
Liu, F., Zheng, L., & Zheng, J. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.04.084 · Cited by 28 · see citing works
Text Representation Using Dependency Tree Subgraphs for Sentiment Analysis
Pak, A., & Paroubek, P. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-20244-5_31 · Cited by 25 · see citing works
Sentiment Lexical-Augmented Convolutional Neural Networks for Sentiment Analysis
Yin, R., Li, P., & Wang, B. (2017) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc.2017.82 · Cited by 17 · see citing works
Audio-Oriented Multimodal Machine Comprehension via Dynamic Inter- and Intra-modality Attention
Huang, Z., Liu, F., Wu, X., Ge, S., Wang, H., Fan, W., & Zou, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17548 · Cited by 16 · see citing works
A text sentiment classification model using double word embedding methods
Zhou, M., Liu, D., Zheng, Y., Zhu, Q., & Guo, P. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09846-x · Cited by 10 · see citing works
Sentiment Analysis for Ad Hoc Discussions Using Multilingual Knowledge-Based Approach
Blekanov, I. S., Kukarkin, M., Maksimov, A., & Bodrunova, S. S. (2018) · International Conference on Applications in Information Technology (ICAIT) · doi:10.1145/3274856.3274880 · Cited by 8 · see citing works
A Hybrid Method for Text-Based Sentiment Analysis
Le, T. V. (2019) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci49370.2019.00260 · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Yang, K.; Xu, H.; Gao, K. CM-BERT: Cross-Modal BERT for Text-Audio Sentiment Analysis. In Proceedings of the 28th ACM International Conference on Multimedia (MM '20), Seattle, WA, USA, 12-16 October 2020; pp. 521-528.
Cited by
Sorted by citations
Multimodal Alignment and Fusion: A Survey
Li, S., & Tang, H. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02667-1 · Cited by 191 · see citing works
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“…model employs specialized encoders (e.g., convolutional neural networks for visual data, transformer architectures for text and audio data) [178, 179] to extract features from different modalities, and then interacts and integrates these features through cross-modal alignment mechanisms [180,…”
SIA-Net: Sparse Interactive Attention Network for Multimodal Emotion Recognition
Li, S., Zhang, T., & Chen, C. L. P. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3409715 · Cited by 14 · see citing works
Hải, N. N., Nguyen, T.-T., & Ngo, Q. T. (2025) · Journal of risk and financial management · doi:10.3390/jrfm18080417 · Cited by 14 · see citing works
He, P., Qi, H., Wang, S., & Cang, J. (2023) · Applied Sciences · doi:10.3390/app13137489 · Cited by 8 · see citing works
Siamese capsule gorilla troops network-based multimodal sentiment analysis for car reviews
Kothuri, S. R., & Rajalakshmi, N. R. (2024) · Soft Computing · doi:10.1007/s00500-024-09754-4 · Cited by 6 · see citing works
Fusion in Multimodal Sentiment Analysis: A Review of Approaches and Challenges
Halim, M. A. M., Zainuddin, N., Samah, K. A. F. A., Mohamad, M., & Selamat, A. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8892-0_30 · Cited by 2 · see citing works
A dual-modality deep learning approach for fine-grained sentiment analysis using speech and text
Agrawal, D., Deshmukh, M., & Singh, S. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.115141 · Cited by 1 · see citing works
Multitask Learning-Based Affective Prediction for Videos of Films and TV Scenes
Su, Z., Lin, S., Zhang, L., Feng, Y., & Jiang, W. (2024) · Applied Sciences · doi:10.3390/app14114391
Text and Speech Adjustable Bimodal Emotion Recognition in Conversational Interface
Rinartha, K., & Ku, L. (2024) · IEEE International Symposium on Consumer Electronics (ISCE) · doi:10.1109/isct62336.2024.10791091
Li, L., Xin, T., & Fang, Z. (2025) · IEEE International Conference of Safe Production and Informatization (IICSPI) · doi:10.1109/iicspi66775.2025.11438177
- Authors
- Zhang, R., Xue, C., Qi, Q., Lin, L., Zhang, J., & Zhang, L.
- Affiliations
- Tianjin University of Science and Technology (CN)
- Gaussian Robotics Pte. Ltd., Tianjin 200100, China
- Corresponding author
- Lin, Liyuan
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2023-02-02
- Volume
- 13
- Issue
- 3
- Pages
- 1915
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4319024667
- Cited by
- 11
- References
- 35
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 4 · 2023: 1
- FWCI
- 0.97
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Tianjin Intelligent Manufacturing Special Fund Project-Underwater Heterogeneous Node Communication and Positioning Integrated Ad Hoc Network System Research and Development (20201207, ZDKT2018-006)
- Tianjin Sino-German University of Applied Sciences Technology Project (20201207, ZDKT2018-006)
- Tianjin Sino-German University of Applied Sciences Video Image Intelligent Analysis and Processing Technology Innovation Team and Tianjin “131” Innovative Talent Team (20201207, ZDKT2018-006)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, cross-modal fusion, multi-head attention, modality alignment, CMU-MOSEI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
