Cited by 231Multimodal Machine Learning ApplicationsSentiment Analysis and Opinion MiningEmotion and Mood Recognition

Multimodal Language Analysis with Recurrent Multistage Fusion

Open access
Liang, Paul Pu, Liu, Ziyin, Bagher Zadeh, AmirAli, Morency, Louis-Philippe·2018·Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

Abstract

Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities.Comprehending multimodal language requires modeling not only the interactions within each modality (intra-modal interactions) but more importantly the interactions between modalities (cross-modal interactions).In this paper, we propose the Recurrent Multistage Fusion Network (RMFN) which decomposes the fusion problem into multiple stages, each of them focused on a subset of multimodal signals for specialized, effective fusion.Crossmodal interactions are modeled using this multistage fusion approach which builds upon intermediate representations of previous stages.Temporal and intra-modal interactions are modeled by integrating our proposed fusion approach with a system of recurrent neural networks.The RMFN displays state-of-the-art performance in modeling human multimodal language across three public datasets relating to multimodal sentiment analysis, emotion recognition, and speaker traits recognition.We provide visualizations to show that each stage of fusion focuses on a different subset of multimodal signals, learning increasingly discriminative multimodal representations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/d18-1014

Cites

72 of 72 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • CMU-MOSEI is the largest dataset of sentence-level sentiment analysis and emotion recognition in real-world online videos.

    “CMU-MOSEI is the largest dataset of sentence-level sentiment analysis and emotion recognition in real-world online videos [102, 183].”

    From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [102]

    The Multimodal Sentiment Analysis section describes CMU-MOSI for sentiment analysis and IEMOCAP for emotion recognition, but does not describe CMU-MOSEI. A search for CMU-MOSEI found no passage establishing its size, sentence-level annotations, or status as the largest dataset. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 7

  1. Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarepa collabo- rative voice analysis repository for speech technolo- gies. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pages 960-964. IEEE.
  2. Sepp Hochreiter and Jürgen Schmidhuber. 1997b. Long short-term memory. Neural Comput., 9(8):1735-1780.
  3. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirec- tional lstm-crf models for sequence tagging. CoRR, abs/1508.01991. · arXiv
  4. Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. 2011. Multi- modal deep learning. In ICML, pages 689-696.
  5. ACL.
  6. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Informa- tion Processing Systems 25, pages 2222-2230.
  7. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. 2016. Re- current Highway Networks. arXiv preprint arXiv:1607.03474. · arXiv

Cited by

Sorted by citations

Authors
Liang, P. P., Liu, Z., Bagher Zadeh, A., & Morency, L.-P.
Affiliations
Carnegie Mellon University (US)
Venue
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Published
2018
Pages
150-161
OpenAlex
W2886193235
Cited by
230
References
69
Citations by year
2026: 12 · 2025: 19 · 2024: 43 · 2023: 37 · 2022: 39 · 2021: 31 · 2020: 32 · 2019: 15 · 2018: 2
FWCI
6.03
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
Funders
  • National Science Foundation
  • Samsung
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
Keywords
cross-modal interaction, recurrent neural networks, multimodal sentiment analysis, emotion recognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review