CoChat

Video sentiment analysis with bimodal information-augmented multi-head attention

Open access
Wu, Ting, Peng, Junjie, Zhang, Wenqiang, Zhang, Huiran, Tan, Shuhua, Yi, Fen, Ma, Chuanshuai, Huang, Yansong · 2022 · Knowledge-Based Systems
Cited by 131

The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.

Abstract

Humans express feelings or emotions via different channels. Take language as an example, it entails different sentiments under different visual-acoustic contexts. To precisely understand human intentions as well as reduce the misunderstandings caused by ambiguity and sarcasm, we should consider multimodal signals including textual, visual and acoustic signals. The crucial challenge is to fuse different modalities of features for sentiment analysis. To effectively fuse the information carried by different modalities and better predict the sentiments, we design a novel multi-head attention based fusion network, which is inspired by the observations that the interactions between any two pair-wise modalities are different and they do not equally contribute to the final sentiment prediction. By assigning the acoustic-visual, acoustic-textual and visual-textual features with reasonable attention and exploiting a residual structure, we attend to attain the significant features. We conduct extensive experiments on four public multimodal datasets including one in Chinese and three in English. The results show that our approach outperforms the existing methods and can explain the contributions of bimodal interaction in multiple modalities.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.knosys.2021.107676

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 26 references

Research works printed but not linked: 36

  1. J. Hu, J. Peng, W. Zhang, L. Qi, H. Zhang, Intention multiple-representation model for logistics intelligent customer service, in: Proc. KSEM Conf., 2020, pp. 186–193. · DOI
  2. Hu, 2021, An intention multiple-representation model with expanded information, Comput. Speech Lang., 68, 10.1016/j.csl.2021.101196 · DOI
  3. Hussain, 2021, Information fusion for affective computing and sentiment analysis, Inf. Fusion, 71, 97, 10.1016/j.inffus.2021.02.010 · DOI
  4. Mai, 2020, Multi-fusion residual memory network for multimodal human sentiment comprehension, IEEE Trans. Affect. Comput.
  5. W. Rahman, M.K. Hasan, S. Lee, A.B. Zadeh, C. Mao, L. Morency, M.E. Hoque, Integrating multimodal information in large pretrained transformers, in: Proc. ACL - Annu. Meet. Assoc. Comput. Linguist. Conf., 2020, pp. 2359–2369. · DOI · Paper page
  6. Huddar, 2020, Multi-level context extraction and attention-based contextual inter-modal fusion for multimodal sentiment analysis and emotion classification, Int. J. Multimedia Inf. Retr., 9, 103, 10.1007/s13735-019-00185-8 · DOI
  7. Majumder, 2018, Multimodal sentiment analysis using hierarchical fusion with context modeling, Knowl. Based Syst., 161, 124, 10.1016/j.knosys.2018.07.041 · DOI · Paper page
  8. W. Yu, H. Xu, F. Meng, Y. Zhu, Y. Ma, J. Wu, J. Zou, K. Yang, CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality, in: Proc. ACL - Annu. Meet. Assoc. Comput. Linguist. Conf., 2020, pp. 3718–3727. · DOI · Paper page
  9. Gkoumas, 2021, What makes the difference? An empirical comparison of fusion strategies for multimodal language analysis, Inf. Fusion, 66, 184, 10.1016/j.inffus.2020.09.005 · DOI
  10. Abdu, 2021, Multimodal video sentiment analysis using deep learning approaches, a survey, Inf. Fusion, 76, 204, 10.1016/j.inffus.2021.06.003 · DOI
  11. R. Li, J. Zhao, J. Hu, S. Guo, Q. Jin, Multi-modal fusion for video sentiment analysis, in: Proc. MuSe - Proc. Int. Multimodal Sentim. Anal. Real-Life Media Chall. Workshop Conf., 2020, pp. 19–25. · DOI
  12. Huddar, 2020, Multi-level feature optimization and multimodal contextual fusion for sentiment analysis and emotion classification, Comput. Intell., 36, 861, 10.1111/coin.12274 · DOI
  13. Mai, 2020, Locally confined modality fusion network with a global perspective for multimodal human affective computing, IEEE Trans. Multimedia, 22, 122, 10.1109/TMM.2019.2925966 · DOI · Paper page
  14. M. Neumann, N.T. Vu, Attentive convolutional neural network based speech emotion recognition: A study on the impact of input features, signal length, and acted speech, in: Proc. Annu. Conf. Int. Speech. Commun. Assoc. Conf., 2017, pp. 1263–1267. · DOI · Paper page
  15. Yu, 2020, Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification, IEEE ACM Trans. Audio Speech Lang. Process., 28, 429, 10.1109/TASLP.2019.2957872 · DOI · Paper page
  16. M. Xu, F. Zhang, S.U. Khan, Improve accuracy of speech emotion recognition with attention head fusion, in: Proc. IEEE - Annu. Comput. Commun. Workshop Conf., 2020, pp. 1058–1064. · DOI
  17. S. Poria, E. Cambria, D. Hazarika, N. Majumder, A. Zadeh, L. Morency, Multi-level multiple attentions for contextual multimodal sentiment analysis, in: Proc. IEEE- Int. Conf. Data Min. Conf., 2017, pp. 1033–1038. · DOI · Paper page
  18. T. Kim, B. Lee, Multi-attention multimodal sentiment analysis, in: Proc. ICMR - Int. Conf. Multimed. Retr. Conf., 2020, pp. 436–441. · DOI
  19. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, in: Proc. Adv. Neural Inf. Proces. Syst. Conf., 2017, pp. 5998–6008.
  20. C. Xi, G. Lu, J. Yan, Multimodal sentiment analysis based on multi-head attention mechanism, in: Proc. ACM - Int. Conf. Proc. Ser. Conf., 2020, pp. 34–39. · DOI · Paper page
  21. Broekens, 2021, Towards transparent robot learning through TDRL-based emotional expressions, IEEE Trans. Affect. Comput., 12, 352, 10.1109/TAFFC.2019.2893348 · DOI
  22. Peng, 2021, Phonetic-enriched text representation for Chinese sentiment analysis with reinforcement learning, Inf. Fusion, 70, 88, 10.1016/j.inffus.2021.01.005 · DOI
  23. Chaturvedi, 2019, Fuzzy commonsense reasoning for multimodal sentiment analysis, Pattern Recognit. Lett., 125, 264, 10.1016/j.patrec.2019.04.024 · DOI · Paper page
  24. Y. Zhang, Z. Wang, J. Du, Deep fusion: An attention guided factorized bilinear pooling for audio-video emotion recognition, in: Proc Int Jt Conf Neural Networks, 2019, pp. 1–8. · DOI
  25. F. Qi, X. Yang, C. Xu, A unified framework for multimodal domain adaptation, in: Proc. ACM - MM Conf., 2018, pp. 429–437. · DOI · Paper page
  26. Stappen, 2021, Sentiment analysis and topic recognition in video transcriptions, IEEE Intell. Syst., 36, 88, 10.1109/MIS.2021.3062200 · DOI · Paper page
  27. Peng, 2021, Adaptive modality distillation for separable multimodal sentiment analysis, IEEE Intell. Syst., 36, 82, 10.1109/MIS.2021.3057757 · DOI
  28. Zhang, 2020, A quantum-like multimodal network framework for modeling interaction dynamics in multiparty conversational sentiment analysis, Inf. Fusion, 62, 14, 10.1016/j.inffus.2020.04.003 · DOI
  29. Zhang, 2018, A quantum-inspired multimodal sentiment analysis framework, Theoret. Comput. Sci., 752, 21, 10.1016/j.tcs.2018.04.029 · DOI · Paper page
  30. Moghimi, 2020, Affective recognition in dynamic and interactive virtual environments, IEEE Trans. Affect. Comput., 11, 45, 10.1109/TAFFC.2017.2764896 · DOI
  31. Noroozi, 2021, Survey on emotional body gesture recognition, IEEE Trans. Affect. Comput., 12, 505, 10.1109/TAFFC.2018.2874986 · DOI · Paper page
  32. B. Mcfee, C. Raffel, D. Liang, D. Ellis, O. Nieto, librosa: Audio and music signal analysis in python, in: Proc. Python in Science Conference, 2015, pp. 18–24. · DOI · Paper page
  33. Y. Wang, Y. Shen, Z. Liu, P.P. Liang, A. Zadeh, L. Morency, Words can shift: Dynamically adjusting word representations using nonverbal behaviors, in: Proc. AAAI - Artif. Intell. Conf., 2019, pp. 7216–7223. · DOI · Paper page
  34. D.S. Chauhan, M.S. Akhtar, A. Ekbal, P. Bhattacharyya, Context-aware interactive attention for multi-modal sentiment and emotion analysis, in: Proc. EMNLP - Empir. Methods Nat. Lang. Process Conf., 2019, pp. 5646–5656. · DOI · Paper page
  35. D. Ghosal, M.S. Akhtar, D.S. Chauhan, S. Poria, A. Ekbal, P. Bhattacharyya, Contextual inter-modal attention for multi-modal sentiment analysis, in: Proc. EMNLP - Empir. Methods Nat. Lang. Process Conf., 2018, pp. 3454–3466. · DOI · Paper page
  36. Van der Maaten, 2008, Visualizing data using t-SNE, J. Mach. Learn. Res., 9, 2579

Cited by

See all 131

Metadata

Authors
Wu, T., Peng, J., Zhang, W., Zhang, H., Tan, S., Yi, F., Ma, C., & Huang, Y.
Venue
Knowledge-Based Systems
Publisher
Elsevier BV
Published
2022-01
Volume
235
Pages
107676
ISSN
  • 2331-8422 (ISSN-L)
  • 0950-7051
OpenAlex
W3208639589
Cited by
131
References
26
Type
posted content
Language
en
Open access
green
Funders
Science and Technology Commission of Shanghai Municipality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Video sentiment analysis with bimodal information-augmented multi-head attention. https://app.cochat.ai/papers/10.1016/j.knosys.2021.107676
The paper
Wu, T., Peng, J., Zhang, W., Zhang, H., Tan, S., Yi, F., Ma, C., & Huang, Y. (2022). Video sentiment analysis with bimodal information-augmented multi-head attention. Knowledge-Based Systems. https://doi.org/10.1016/j.knosys.2021.107676

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.