Video sentiment analysis with bimodal information-augmented multi-head attention
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Humans express feelings or emotions via different channels. Take language as an example, it entails different sentiments under different visual-acoustic contexts. To precisely understand human intentions as well as reduce the misunderstandings caused by ambiguity and sarcasm, we should consider multimodal signals including textual, visual and acoustic signals. The crucial challenge is to fuse different modalities of features for sentiment analysis. To effectively fuse the information carried by different modalities and better predict the sentiments, we design a novel multi-head attention based fusion network, which is inspired by the observations that the interactions between any two pair-wise modalities are different and they do not equally contribute to the final sentiment prediction. By assigning the acoustic-visual, acoustic-textual and visual-textual features with reasonable attention and exploiting a residual structure, we attend to attain the significant features. We conduct extensive experiments on four public multimodal datasets including one in Chinese and three in English. The results show that our approach outperforms the existing methods and can explain the contributions of bimodal interaction in multiple modalities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259 · cited 4×
“The MulT [41] uses its cross-modal attention module to extract the key information inside each modality and then merges these features based on the Transformer [45] model.”
BackgroundMethodInfluentialTensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 18×
“Some popular methods such as Tensor Fusion Network (TFN) [10], Memory Fusion Network (MFN) [13], Dynamic Fusion Network (DFN) [14] try to capture the interaction relations between modalities dynamically.”
BackgroundMethodInfluentialMultimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · cited 1×
“Some popular methods such as Tensor Fusion Network (TFN) [10], Memory Fusion Network (MFN) [13], Dynamic Fusion Network (DFN) [14] try to capture the interaction relations between modalities dynamically.”
Method
Research works printed but not linked: 36
- J. Hu, J. Peng, W. Zhang, L. Qi, H. Zhang, Intention multiple-representation model for logistics intelligent customer service, in: Proc. KSEM Conf., 2020, pp. 186–193. · DOI
- Hu, 2021, An intention multiple-representation model with expanded information, Comput. Speech Lang., 68, 10.1016/j.csl.2021.101196 · DOI
- Hussain, 2021, Information fusion for affective computing and sentiment analysis, Inf. Fusion, 71, 97, 10.1016/j.inffus.2021.02.010 · DOI
- Mai, 2020, Multi-fusion residual memory network for multimodal human sentiment comprehension, IEEE Trans. Affect. Comput.
- W. Rahman, M.K. Hasan, S. Lee, A.B. Zadeh, C. Mao, L. Morency, M.E. Hoque, Integrating multimodal information in large pretrained transformers, in: Proc. ACL - Annu. Meet. Assoc. Comput. Linguist. Conf., 2020, pp. 2359–2369. · DOI · Paper page
- Huddar, 2020, Multi-level context extraction and attention-based contextual inter-modal fusion for multimodal sentiment analysis and emotion classification, Int. J. Multimedia Inf. Retr., 9, 103, 10.1007/s13735-019-00185-8 · DOI
- Majumder, 2018, Multimodal sentiment analysis using hierarchical fusion with context modeling, Knowl. Based Syst., 161, 124, 10.1016/j.knosys.2018.07.041 · DOI · Paper page
- W. Yu, H. Xu, F. Meng, Y. Zhu, Y. Ma, J. Wu, J. Zou, K. Yang, CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality, in: Proc. ACL - Annu. Meet. Assoc. Comput. Linguist. Conf., 2020, pp. 3718–3727. · DOI · Paper page
- Gkoumas, 2021, What makes the difference? An empirical comparison of fusion strategies for multimodal language analysis, Inf. Fusion, 66, 184, 10.1016/j.inffus.2020.09.005 · DOI
- Abdu, 2021, Multimodal video sentiment analysis using deep learning approaches, a survey, Inf. Fusion, 76, 204, 10.1016/j.inffus.2021.06.003 · DOI
- R. Li, J. Zhao, J. Hu, S. Guo, Q. Jin, Multi-modal fusion for video sentiment analysis, in: Proc. MuSe - Proc. Int. Multimodal Sentim. Anal. Real-Life Media Chall. Workshop Conf., 2020, pp. 19–25. · DOI
- Huddar, 2020, Multi-level feature optimization and multimodal contextual fusion for sentiment analysis and emotion classification, Comput. Intell., 36, 861, 10.1111/coin.12274 · DOI
- Mai, 2020, Locally confined modality fusion network with a global perspective for multimodal human affective computing, IEEE Trans. Multimedia, 22, 122, 10.1109/TMM.2019.2925966 · DOI · Paper page
- M. Neumann, N.T. Vu, Attentive convolutional neural network based speech emotion recognition: A study on the impact of input features, signal length, and acted speech, in: Proc. Annu. Conf. Int. Speech. Commun. Assoc. Conf., 2017, pp. 1263–1267. · DOI · Paper page
- Yu, 2020, Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification, IEEE ACM Trans. Audio Speech Lang. Process., 28, 429, 10.1109/TASLP.2019.2957872 · DOI · Paper page
- M. Xu, F. Zhang, S.U. Khan, Improve accuracy of speech emotion recognition with attention head fusion, in: Proc. IEEE - Annu. Comput. Commun. Workshop Conf., 2020, pp. 1058–1064. · DOI
- S. Poria, E. Cambria, D. Hazarika, N. Majumder, A. Zadeh, L. Morency, Multi-level multiple attentions for contextual multimodal sentiment analysis, in: Proc. IEEE- Int. Conf. Data Min. Conf., 2017, pp. 1033–1038. · DOI · Paper page
- T. Kim, B. Lee, Multi-attention multimodal sentiment analysis, in: Proc. ICMR - Int. Conf. Multimed. Retr. Conf., 2020, pp. 436–441. · DOI
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, in: Proc. Adv. Neural Inf. Proces. Syst. Conf., 2017, pp. 5998–6008.
- C. Xi, G. Lu, J. Yan, Multimodal sentiment analysis based on multi-head attention mechanism, in: Proc. ACM - Int. Conf. Proc. Ser. Conf., 2020, pp. 34–39. · DOI · Paper page
- Broekens, 2021, Towards transparent robot learning through TDRL-based emotional expressions, IEEE Trans. Affect. Comput., 12, 352, 10.1109/TAFFC.2019.2893348 · DOI
- Peng, 2021, Phonetic-enriched text representation for Chinese sentiment analysis with reinforcement learning, Inf. Fusion, 70, 88, 10.1016/j.inffus.2021.01.005 · DOI
- Chaturvedi, 2019, Fuzzy commonsense reasoning for multimodal sentiment analysis, Pattern Recognit. Lett., 125, 264, 10.1016/j.patrec.2019.04.024 · DOI · Paper page
- Y. Zhang, Z. Wang, J. Du, Deep fusion: An attention guided factorized bilinear pooling for audio-video emotion recognition, in: Proc Int Jt Conf Neural Networks, 2019, pp. 1–8. · DOI
- F. Qi, X. Yang, C. Xu, A unified framework for multimodal domain adaptation, in: Proc. ACM - MM Conf., 2018, pp. 429–437. · DOI · Paper page
- Stappen, 2021, Sentiment analysis and topic recognition in video transcriptions, IEEE Intell. Syst., 36, 88, 10.1109/MIS.2021.3062200 · DOI · Paper page
- Peng, 2021, Adaptive modality distillation for separable multimodal sentiment analysis, IEEE Intell. Syst., 36, 82, 10.1109/MIS.2021.3057757 · DOI
- Zhang, 2020, A quantum-like multimodal network framework for modeling interaction dynamics in multiparty conversational sentiment analysis, Inf. Fusion, 62, 14, 10.1016/j.inffus.2020.04.003 · DOI
- Zhang, 2018, A quantum-inspired multimodal sentiment analysis framework, Theoret. Comput. Sci., 752, 21, 10.1016/j.tcs.2018.04.029 · DOI · Paper page
- Moghimi, 2020, Affective recognition in dynamic and interactive virtual environments, IEEE Trans. Affect. Comput., 11, 45, 10.1109/TAFFC.2017.2764896 · DOI
- Noroozi, 2021, Survey on emotional body gesture recognition, IEEE Trans. Affect. Comput., 12, 505, 10.1109/TAFFC.2018.2874986 · DOI · Paper page
- B. Mcfee, C. Raffel, D. Liang, D. Ellis, O. Nieto, librosa: Audio and music signal analysis in python, in: Proc. Python in Science Conference, 2015, pp. 18–24. · DOI · Paper page
- Y. Wang, Y. Shen, Z. Liu, P.P. Liang, A. Zadeh, L. Morency, Words can shift: Dynamically adjusting word representations using nonverbal behaviors, in: Proc. AAAI - Artif. Intell. Conf., 2019, pp. 7216–7223. · DOI · Paper page
- D.S. Chauhan, M.S. Akhtar, A. Ekbal, P. Bhattacharyya, Context-aware interactive attention for multi-modal sentiment and emotion analysis, in: Proc. EMNLP - Empir. Methods Nat. Lang. Process Conf., 2019, pp. 5646–5656. · DOI · Paper page
- D. Ghosal, M.S. Akhtar, D.S. Chauhan, S. Poria, A. Ekbal, P. Bhattacharyya, Contextual inter-modal attention for multi-modal sentiment analysis, in: Proc. EMNLP - Empir. Methods Nat. Lang. Process Conf., 2018, pp. 3454–3466. · DOI · Paper page
- Van der Maaten, 2008, Visualizing data using t-SNE, J. Mach. Learn. Res., 9, 2579
Cited by
Gandhi, A., Adhvaryu, K., Poria, S., Cambria, E., & Hussain, A. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.09.025 · Cited by 783
“[39] BIMHA 2022 Bimodal Fusion CH-SIMS Bimodal Information-augmented Multi-Head Attention Inter-modal interaction and inter-bimodal interaction 82.”
Recent advancements and challenges of NLP-based sentiment analysis: A state-of-the-art review
Jim, J. R., Talukder, M. A. R., Malakar, P., Kabir, M. M., Nur, K., & Mridha, M. F. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100059 · Cited by 297
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“Additionally, Wu et al. [106] developed a fusion network named Bimodal Information-augmented Multi-Head Attention (BIMHA), which relies on multi-head attention and comprises a total of four layers.”
BackgroundSentiment Analysis in the Age of Generative AI
Krugmann, J. O., & Hartmann, J. (2024) · Customer Needs and Solutions · doi:10.1007/s40547-024-00143-4 · Cited by 189
Natha, S., Siraj, M., Ahmed, F., Altamimi, M., & Syed, M. (2025) · IEEE Access · doi:10.1109/access.2025.3574835 · Cited by 22
“There are various attention techniques developed for anomaly detection models including Self-Attention (SA), Bidirectional LSTM (BiLSTM) [25], and Multi-Head Self-Attention (MHSA) [26].”
Metadata
- Authors
- Wu, T., Peng, J., Zhang, W., Zhang, H., Tan, S., Yi, F., Ma, C., & Huang, Y.
- Venue
- Knowledge-Based Systems
- Publisher
- Elsevier BV
- Published
- 2022-01
- Volume
- 235
- Pages
- 107676
- ISSN
- 2331-8422 (ISSN-L)
- 0950-7051
- OpenAlex
- W3208639589
- Cited by
- 131
- References
- 26
- Type
- posted content
- Language
- en
- Open access
- green
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
- Funders
- Science and Technology Commission of Shanghai Municipality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Video sentiment analysis with bimodal information-augmented multi-head attention. https://app.cochat.ai/papers/10.1016/j.knosys.2021.107676
- The paper
- Wu, T., Peng, J., Zhang, W., Zhang, H., Tan, S., Yi, F., Ma, C., & Huang, Y. (2022). Video sentiment analysis with bimodal information-augmented multi-head attention. Knowledge-Based Systems. https://doi.org/10.1016/j.knosys.2021.107676
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.