Tensor Fusion Network for Multimodal Sentiment Analysis
Open accessAbstract
Multimodal sentiment analysis is an increasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a multimodal setup where other relevant modalities accompany language.In this paper, we pose the problem of multimodal sentiment analysis as modeling intra-modality and inter-modality dynamics.We introduce a novel model, termed Tensor Fusion Network, which learns both such dynamics end-to-end.The proposed approach is tailored for the volatile nature of spoken language in online videos as well as accompanying gestures and voice.In the experiments, our model outperforms state-ofthe-art approaches for both multimodal and unimodal sentiment analysis.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Tensors are specifically designed to tackle the multimodal complementarity challenge by explicitly capturing higher-order interactions across modalities.
“Tensors are specifically designed to tackle the multimodal complementarity challenge by explicitly capturing higher-order interactions across modalities [179].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [179]
Inter-modality dynamics is modeled with Tensor Fusion, which explicitly aggregates unimodal, bimodal and trimodal interactions.
“Inter-modality dynamics is modeled with a new multimodal fusion approach, named Tensor Fusion, which explicitly aggregates unimodal, bimodal and trimodal interactions.”
From Survey on AI-Based Multimodal Methods for Emotion Detection (2019) · cites this paper as [9]
Intra-modality dynamics is modeled through three Modality Embedding Subnetworks for language, visual and acoustic modalities.
“Intra-modality dynamics is modeled through three Modality Embedding Subnetworks, for language, visual and acoustic modalities, respectively [9].”
From Survey on AI-Based Multimodal Methods for Emotion Detection (2019) · cites this paper as [9]
Zadeh introduced the Tensor Fusion Network (TFN), which combines intra-modality and inter-modality models.
“To solve this problem, Zadeh introduced the Tensor Fusion Network (TFN), which combine the intra-modality and inter-modality models.”
From Survey on AI-Based Multimodal Methods for Emotion Detection (2019) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 41 · Cited by 2,102
Research works printed but not linked: 15
- Basant Agarwal, Soujanya Poria, Namita Mittal, Alexander Gelbukh, and Amir Hussain. 2015. Concept-level sentiment analysis with dependency- based semantic parsing: a novel approach. Cogni- tive Computation 7(4):487-499.
- Paavo Alku. 1992. Glottal wave analysis with pitch synchronous iterative adaptive inverse filtering. Speech communication 11(2-3):109-118.
- Paavo Alku, Tom Bäckström, and Erkki Vilkman. 2002. Normalized amplitude quotient for parametrization of the glottal flow. the Journal of the Acoustical So- ciety of America 112(2):701-710.
- Paavo Alku, Helmer Strik, and Erkki Vilkman. 1997. Parabolic spectral parameter-a new method for quantification of the glottal flow. Speech Commu- nication 22(1):67-79.
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question an- swering. In Proceedings of the IEEE International Conference on Computer Vision. pages 2425-2433.
- Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep-a col- laborative voice analysis repository for speech tech- nologies. In Acoustics, Speech and Signal Process- ing (ICASSP), 2014 IEEE International Conference on. IEEE, pages 960-964.
- Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion 6(3-4):169-200.
- Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with lstm. Neural computation 12(10):2451-2471.
- Sayan Ghosh, Eugene Laksana, Louis-Philippe Morency, and Stefan Scherer. 2016b. Represen- tation learning for speech emotion recognition. Interspeech 2016 pages 3603-3607.
- Michael Glodek, Stephan Tschechne, Georg Lay- her, Martin Schels, Tobias Brosch, Stefan Scherer, Markus Kächele, Miriam Schmidt, Heiko Neumann, Günther Palm, et al. 2011. Multiple classifier sys- tems for the classification of audio-visual emotional states. In Affective Computing and Intelligent Inter- action, Springer, pages 359-368.
- Mohit Iyyer, Varun Manjunatha, Jordan L Boyd- Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classi- fication. In ACL (1). pages 1681-1691.
- Soujanya Poria, Basant Agarwal, Alexander Gel- bukh, Amir Hussain, and Newton Howard. 2014a. Dependency-based semantic parsing for concept- level text analysis. In International Conference on Intelligent Text Processing and Computational Lin- guistics. Springer, pages 113-127.
- Michel Valstar, Jonathan Gratch, Björn Schuller, Fa- bien Ringeval, Dennis Lalanne, Mercedes Tor- res Torres, Stefan Scherer, Giota Stratou, Roddy Cowie, and Maja Pantic. 2016. Avec 2016: De- pression, mood, and emotion recognition workshop and challenge. In Proceedings of the 6th Inter- national Workshop on Audio/Visual Emotion Chal- lenge. ACM, pages 3-10.
- Martin Wöllmer, Felix Weninger, Tobias Knaup, Björn Schuller, Congkai Sun, Kenji Sagae, and Louis- Philippe Morency. 2013. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems 28(3):46-53.
- Bishan Yang and Claire Cardie. 2012. Extracting opin- ion expressions with semi-markov conditional ran- dom fields. In Proceedings of the 2012 Joint Con- ference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguis- tics, pages 1335-1345.
- Authors
- Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- University of Stirling (GB)
- National University of Singapore (SG)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 1103-1114
- OpenAlex
- W2738581557
- Cited by
- 2,102
- References
- 41
- Citations by year
- 2026: 364 · 2025: 428 · 2024: 358 · 2023: 283 · 2022: 217 · 2021: 174 · 2020: 139 · 2019: 88 · 2018: 46 · 2017: 5
- FWCI
- 34.67
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Keywords
- multimodal sentiment analysis, end-to-end learning, gesture, voice, spoken language
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
