CoChat
Sign Up
Cited by 50Emotion and Mood RecognitionSpeech and dialogue systemsSentiment Analysis and Opinion Mining

DialogueTRM: Exploring Multi-Modal Emotional Dynamics in a Conversation

Open access
Mao, Yuzhao, Liu, Guang, Wang, Xiaojie, Gao, Weiguo, Li, Xuan·2021·Findings of the Association for Computational Linguistics: EMNLP 2021
DOI 10.18653/v1/2021.findings-emnlp.229
Open this paper in CoChat — free accountOpen PDF

Abstract

Emotion dynamics formulates principles explaining the emotional fluctuation during conversations.Recent studies explore the emotion dynamics from the self and inter-personal dependencies, however, ignoring the temporal and spatial dependencies in the situation of multi-modal conversations.To address the issue, we extend the concept of emotion dynamics to multi-modal settings and propose a Dialogue Transformer for simultaneously modeling the intra-modal and inter-modal emotion dynamics.Specifically, the intra-modal emotion dynamics is to not only capture the temporal dependency but also satisfy the context preference in every single modality.The intermodal emotional dynamics aims at handling multi-grained spatial dependency across all modalities.Our models outperform the stateof-the-art with a margin of 4%-16% for most of the metrics on three benchmark datasets.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.findings-emnlp.229

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most previous MERMC studies primarily focus on the text and audio modalities.

    “In the literature, most previous MERMC studies primarily focus on the text and audio modalities (Poria et al., 2019; Liang et al., 2020; Mao et al., 2021; Chen et al., 2021)”

    From A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations (2023) · cites this paper as Mao et al., 2021

    The Related Work section discusses earlier emotion-recognition studies, including recent work using all three modalities, but does not say whether most previous MERMC studies focused primarily on text and audio. Searches for previous studies’ use of text, acoustic, and visual modalities did not establish the claimed majority. (LLM, conf=0.69)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 60 · Cited by 50

Research works printed but not linked: 49

  1. Sujay Angadi and R Venkata Siva Reddy. 2019. Sur- vey on sentiment analysis from affective multimodal content. In Smart intelligent computing and applica- tions, pages 599-607. Springer.
  2. John Arevalo, Thamar Solorio, Manuel Montes-y Gomez, and Fabio A González. 2020. Gated mul- timodal networks. Neural Computing and Applica- tions, pages 1-20.
  3. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language re- sources and evaluation, 42(4):335.
  4. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. · arXiv
  5. Niko Colneriĉ and Janez Demsar. 2018. Emotion recognition on twitter: Comparative study and train- ing a unison model. IEEE transactions on affective computing.
  6. Dragos Datcu and Leon JM Rothkrantz. 2014. Seman- tic audio-visual data fusion for automatic emotion recognition. Emotion recognition: a pattern analy- sis approach, pages 411-435.
  7. Paul Ekman. 1993. Facial expression and emotion. American psychologist, 48(4):384.
  8. Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multi- media, pages 1459-1462.
  9. Fangxiang Feng, Xiaojie Wang, and Ruifan Li. 2014. Cross-modal retrieval with correspondence autoen- coder. In Proceedings of the 22nd ACM interna- tional conference on Multimedia, pages 7-16.
  10. Andrea Frome, Greg S Corrado, Jon Shlens, Samy Ben- gio, Jeff Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. 2013. Devise: A deep visual-semantic em- bedding model. In Advances in neural information processing systems, pages 2121-2129.
  11. A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. 2016. Multimodal compact bilin- ear pooling for visual question answering and visual grounding. In EMNLP.
  12. Deepanway Ghosal, Navonil Majumder, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria. 2020. COSMIC: COmmonSense knowledge for eMotion identification in conversations. In Findings of EMNLP.
  13. Deepanway Ghosal, Navonil Majumder, Soujanya Po- ria, Niyati Chhaya, and Alexander Gelbukh. 2019. DialogueGCN: A graph convolutional neural net- work for emotion recognition in conversation. In EMNLP-IJCNLP, Hong Kong, China. Association for Computational Linguistics.
  14. Yue Gu, Xinyu Lyu, Weijia Sun, Weitian Li, Shuhong Chen, Xinyu Li, and Ivan Marsic. 2019. Mutual cor- relation attentive factors in dyadic fusion networks for speech emotion recognition. In Proceedings of the 27th ACM International Conference on Multime- dia, pages 157-166.
  15. Wenzhong Guo, Jianwen Wang, and Shiping Wang. 2019. Deep multimodal representation learning: A survey. IEEE Access, 7:63373-63394.
  16. Devamanyu Hazarika, Soujanya Poria, Rada Mihal- cea, Erik Cambria, and Roger Zimmermann. 2018a. Icon: interactive conversational memory network for multimodal emotion detection. In Proceedings of the 2018 Conference on Empirical Methods in Natu- ral Language Processing, pages 2594-2604.
  17. Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018b. Conversational memory net- work for emotion recognition in dyadic dialogue videos. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2122- 2132.
  18. Devamanyu Hazarika, Soujanya Poria, Roger Zimmer- mann, and Rada Mihalcea. 2019. Emotion recog- nition in conversations with transfer learning from generative conversation modeling. arXiv preprint arXiv:1910.04980. · arXiv
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
  20. Chao-Chun Hsu, Sheng-Yeh Chen, Chuan-Chun Kuo, Ting-Hao Huang, and Lun-Wei Ku. 2018. Emotion- lines: An emotion corpus of multi-party conversa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  21. Wenxiang Jiao, Michael R Lyu, and Irwin King. 2019. Real-time emotion recognition via attention gated hierarchical memory network. arXiv preprint arXiv:1911.09075. · arXiv
  22. Song Kaitao, Tan Xu, Qin Tao, Lu Jianfeng, and Liu Tie-yan. 2019. Mass:masked sequence to sequence pre-training for language generation. In ICML.
  23. Ramandeep Kaur and Sandeep Kautish. 2019. Mul- timodal sentiment analysis: A survey and compari- son. International Journal of Service Science, Man- agement, Engineering, and Technology (IJSSMET), 10(2):38-58.
  24. Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal neural language models. In In- ternational conference on machine learning, pages 595-603.
  25. Bernhard Kratzwald, Suzana Ilic, Mathias Kraus, Ste- fan Feuerriegel, and Helmut Prendinger. 2018. De- cision support with text-based emotion recogni- tion: Deep learning for affective computing. arXiv preprint arXiv:1803.06397. · arXiv
  26. Wei Li, Wei Shao, Shaoxiong Ji, and Erik Cambria. 2020. Bieru: Bidirectional emotional recurrent unit for conversational sentiment analysis. CoRR.
  27. I Loshchilov and F Hutter. Fixing weight decay regular- ization in adam, corr, abs/1711.05101. In Proceed- ings of the ICLR 2018 Conference Blind Submission, Vancouver, BC, Canada, volume 30. · arXiv
  28. Navonil Majumder, Soujanya Poria, Devamanyu Haz- arika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6818-6825.
  29. Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. 2014. Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632. · arXiv
  30. Yuzhao Mao, Chang Zhou, Xiaojie Wang, and Ruifan Li. 2018. Show and tell more: Topic-oriented multi- sentence image captioning. In IJCAI, pages 4258- 4264.
  31. Saif M Mohammad and Peter D Turney. 2010. Emo- tions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In Proceedings of the NAACL HLT 2010 workshop on computational approaches to analysis and genera- tion of emotion in text, pages 26-34. Association for Computational Linguistics.
  32. Rosalind W Picard. 2010. Affective computing: from laughter to ieee. IEEE Transactions on Affective Computing, 1(1):11-17.
  33. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017. Context-dependent sentiment anal- ysis in user-generated videos. In Proceedings of the 55th annual meeting of the association for compu- tational linguistics (volume 1: Long papers), pages 873-883.
  34. Soujanya Poria, Devamanyu Hazarika, Navonil Ma- jumder, Gautam Naik, Erik Cambria, and Rada Mi- halcea. 2019a. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 527- 536.
  35. Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019b. Emotion recognition in conversation: Research challenges, datasets, and re- cent advances. IEEE Access, 7:100943-100953.
  36. Marc Schröder. 2003. Experimental study of affect bursts. Speech communication, 40(1-2):99-116.
  37. Björn Schuller, Michel Valster, Florian Eyben, Roddy Cowie, and Maja Pantic. 2012. Avec 2012: the con- tinuous audio/visual emotion challenge. In Proceed- ings of the 14th ACM international conference on Multimodal interaction, pages 449-456.
  38. Shadi Shaheen, Wassim El-Hajj, Hazem Hajj, and Shady Elbassuoni. 2014. Emotion recognition from text based on automatically generated rules. In 2014 IEEE International Conference on Data Min- ing Workshop, pages 383-392. IEEE.
  39. Richard Socher, Milind Ganjoo, Christopher D Man- ning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In Advances in neu- ral information processing systems, pages 935-943.
  40. Richard Socher, Andrej Karpathy, Quoc V Le, Christo- pher D Manning, and Andrew Y Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2:207-218.
  41. Nitish Srivastava and Russ R Salakhutdinov. 2012. Multimodal learning with deep boltzmann machines. In Advances in neural information processing sys- tems, pages 2222-2230.
  42. Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Tor- resani, and Manohar Paluri. 2015. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 4489-4497.
  43. Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019b. Learning factorized multimodal representa- tions. In International Conference on Representa- tion Learning.
  44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages 5998-6008.
  45. Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2019. Words can shift: Dynamically adjusting word repre- sentations using nonverbal behaviors. In Proceed- ings of the AAAI Conference on Artificial Intelli- gence, volume 33, pages 7216-7223.
  46. Chenfei Wu, Jinlai Liu, Xiaojie Wang, and Ruifan Li. 2019. Differential networks for visual question an- swering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8997-9004.
  47. Chung-Hsien Wu, Ze-Jing Chuang, and Yu-Chung Lin. 2006. Emotion recognition from text using seman- tic labels and separable mixture models. ACM trans- actions on Asian language information processing (TALIP), 5(2):165-183.
  48. Zhihong Zeng, Yuxiao Hu, Glenn I Roisman, Zhen Wen, Yun Fu, and Thomas S Huang. 2007. Audio- visual spontaneous emotion recognition. In Artifi- cal intelligence for human computing, pages 72-90. Springer.
  49. Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018. Emotional chatting ma- chine: Emotional conversation generation with in- ternal and external memory. In Thirty-Second AAAI Conference on Artificial Intelligence.

Authors
Mao, Y., Liu, G., Wang, X., Gao, W., & Li, X.
Affiliations
Beijing University of Posts and Telecommunications (CN)
Corresponding author
Mao, Yuzhao
ORCID
Venue
Findings of the Association for Computational Linguistics: EMNLP 2021
Publisher
Association for Computational Linguistics
Published
2021
Pages
2694-2704
DOI
10.18653/v1/2021.findings-emnlp.229
OpenAlex
W3211488063
Cited by
50
References
60
Citations by year
2026: 13 · 2025: 16 · 2024: 7 · 2023: 11 · 2022: 2 · 2021: 1
FWCI
5.84
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
emotional dynamics, temporal dependencies, spatial dependence, mood fluctuations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review