CoChat
Sign Up
Cited by 108Emotion and Mood RecognitionSentiment Analysis and Opinion MiningSpeech and dialogue systems

Robust Multimodal Emotion Recognition from Conversation with Transformer-Based Crossmodality Fusion

Open access
Xie, Baijun, Sidulova, Mariia, Park, Chung Hyuk·2021·Sensors
DOI 10.3390/s21144913PMC PMC8309929
Open this paper in CoChat — free accountOpen PDF

Abstract

Decades of scientific research have been conducted on developing and evaluating methods for automated emotion recognition. With exponentially growing technology, there is a wide range of emerging applications that require emotional state recognition of the user. This paper investigates a robust approach for multimodal emotion recognition during a conversation. Three separate models for audio, video and text modalities are structured and fine-tuned on the MELD. In this paper, a transformer-based crossmodality fusion with the EmbraceNet architecture is employed to estimate the emotion. The proposed multimodal network architecture can achieve up to 65% accuracy, which significantly surpasses any of the unimodal models. We provide multiple evaluation techniques applied to our work to show that our model is robust and can even outperform the state-of-the-art models on the MELD.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/s21144913

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. [1]Han K. Yu D. Tashev I. Speech emotion recognition using deep neural network and extreme learning machine Proceedings of the Fifteenth Annual Conference of the International Speech Communication Association Singapore 14–18 September 2014
  2. [2]Chen M. He X. Yang J. Zhang H. 3-D convolutional recurrent neural networks with attention model for speech emotion recognition IEEE Signal Process. Lett. 2018 25 1440 1444 10.1109/LSP.2018.2860246 · DOI · Paper page
  3. [3]Nediyanchath A. Paramasivam P. Yenigalla P. Multi-head attention for speech emotion recognition with auxiliary learning of gender recognition Proceedings of the ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) Barcelona, Spain 4–8 May 2020 7179 7183
  4. [4]Chatterjee A. Gupta U. Chinnakotla M.K. Srikanth R. Galley M. Agrawal P. Understanding emotions in text using deep learning and big data Comput. Hum. Behav. 2019 93 309 317 10.1016/j.chb.2018.12.029 · DOI
  5. [5]Batbaatar E. Li M. Ryu K.H. Semantic-emotion neural network for emotion recognition from text IEEE Access 2019 7 111866 111878 10.1109/ACCESS.2019.2934529 · DOI
  6. [6]Tarnowski P. Kołodziej M. Majkowski A. Rak R.J. Emotion recognition using facial expressions Procedia Comput. Sci. 2017 108 1175 1184 10.1016/j.procs.2017.05.025 · DOI
  7. [7]Cohen I. Garg A. Huang T.S. Emotion recognition from facial expressions using multilevel HMM Neural Information Processing Systems Citeseer State College, PA, USA 2000 Volume 2
  8. [8]Regenbogen C. Schneider D.A. Finkelmeyer A. Kohn N. Derntl B. Kellermann T. Gur R.E. Schneider F. Habel U. The differential contribution of facial expressions, prosody, and speech content to empathy Cogn. Emot. 2012 26 995 1014 10.1080/02699931.2011.631296 22214265 · DOI
  9. [9]Regenbogen C. Schneider D.A. Gur R.E. Schneider F. Habel U. Kellermann T. Multimodal human communication—Targeting facial expressions, speech content and prosody Neuroimage 2012 60 2346 2356 10.1016/j.neuroimage.2012.02.043 22487549 · DOI
  10. [10]Jessen S. Kotz S.A. The temporal dynamics of processing emotions from vocal, facial, and bodily expressions Neuroimage 2011 58 665 674 10.1016/j.neuroimage.2011.06.035 21718792 · DOI
  11. [11]Müller V.I. Habel U. Derntl B. Schneider F. Zilles K. Turetsky B.I. Eickhoff S.B. Incongruence effects in crossmodal emotional integration Neuroimage 2011 54 2257 2266 10.1016/j.neuroimage.2010.10.047 20974266 PMC8007888 · DOI
  12. [12]Stiefelhagen R. Ekenel H.K. Fugen C. Gieselmann P. Holzapfel H. Kraft F. Nickel K. Voit M. Waibel A. Enabling multimodal human–robot interaction for the karlsruhe humanoid robot IEEE Trans. Robot. 2007 23 840 851 10.1109/TRO.2007.907484 · DOI
  13. [13]Hong A. Lunscher N. Hu T. Tsuboi Y. Zhang X. dos Reis Alves S.F. Nejat G. Benhabib B. A Multimodal Emotional Human-Robot Interaction Architecture for Social Robots Engaged in Bi-directional Communication IEEE Trans. Cybern. 2020 10.1109/TCYB.2020.2974688 32149676 · DOI
  14. [14]Kim J.C. Azzi P. Jeon M. Howard A.M. Park C.H. Audio-based emotion estimation for interactive robotic therapy for children with autism spectrum disorder Proceedings of the 2017 IEEE 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI) Jeju, Korea 28 June–1 July 2017 39 44
  15. [15]Xie B. Kim J.C. Park C.H. Musical emotion recognition with spectral feature extraction based on a sinusoidal model with model-based and deep-learning approaches Appl. Sci. 2020 10 902 10.3390/app10030902 PMC9109831 35582331 · DOI
  16. [16]Maat L. Pantic M. Gaze-X: Adaptive, affective, multimodal interface for single-user office scenarios Artifical Intelligence for Human Computing Springer Berlin/Heidelberg, Germany 2007 251 271
  17. [17]Kapoor A. Burleson W. Picard R.W. Automatic prediction of frustration Int. J. Hum. Comput. Stud. 2007 65 724 736 10.1016/j.ijhcs.2007.02.003 · DOI · Paper page
  18. [18]Murray I.R. Arnott J.L. Synthesizing emotions in speech: Is it time to get excited? Proceedings of the IEEE Fourth International Conference on Spoken Language Processing (ICSLP’96) Philadelphia, PA, USA 3–6 October 1996 Volume 3 1816 1819
  19. [19]Walker M.A. Cahn J.E. Whittaker S.J. Improvising linguistic style: Social and affective bases for agent personality Proceedings of the First International Conference on Autonomous Agents Marina Del Rey, CA, USA 5–8 February 1997 96 105
  20. [20]Schröder M. Emotional speech synthesis: A review Proceedings of the Seventh European Conference on Speech Communication and Technology Aalborg, Denmark 3–7 September 2001
  21. [22]Edwards J. Jackson H.J. Pattison P.E. Emotion recognition via facial expression and affective prosody in schizophrenia: A methodological review Clin. Psychol. Rev. 2002 22 789 832 10.1016/S0272-7358(02)00130-7 12214327 · DOI
  22. [23]Streit M. Wölwer W. Gaebel W. Facial-affect recognition and visual scanning behaviour in the course of schizophrenia Schizophr. Res. 1997 24 311 317 10.1016/S0920-9964(96)00126-0 9134591 · DOI
  23. [24]Sebe N. Cohen I. Huang T.S. Multimodal emotion recognition Handbook of Pattern Recognition and Computer Vision World Scientific Singapore 2005 387 409
  24. [25]Kessous L. Castellano G. Caridakis G. Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis J. Multimodal User Interfaces 2010 3 33 48 10.1007/s12193-009-0025-5 · DOI · Paper page
  25. [26]Samani H.A. Saadatian E. A multidisciplinary artificial intelligence model of an affective robot Int. J. Adv. Robot. Syst. 2012 9 6 10.5772/45662 · DOI
  26. [27]Barros P. Magg S. Weber C. Wermter S. A multichannel convolutional neural network for hand posture recognition International Conference on Artificial Neural Networks Springer Berlin/Heidelberg, Germany 2014 403 410
  27. [28]Javed H. Lee W. Park C.H. Toward an Automated Measure of Social Engagement for Children With Autism Spectrum Disorder—A Personalized Computational Modeling Approach Front. Robot. AI 2020 7 43 10.3389/frobt.2020.00043 33501211 PMC7805713 · DOI
  28. [29]Wu C.H. Lin J.C. Wei W.L. Survey on audiovisual emotion recognition: Databases, features, and data fusion strategies APSIPA Trans. Signal Inf. Process. 2014 3 e12 10.1017/ATSIP.2014.11 · DOI
  29. [30]Schuller B. Valster M. Eyben F. Cowie R. Pantic M. Avec 2012: The continuous audio/visual emotion challenge Proceedings of the 14th ACM International Conference on Multimodal Interaction Santa Monica, CA, USA 22–26 October 2012 449 456
  30. [31]Huang J. Li Y. Tao J. Lian Z. Wen Z. Yang M. Yi J. Continuous multimodal emotion prediction based on long short term memory recurrent neural network Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge Mountain View, CA, USA 23–27 October 2017 11 18
  31. [32]Liu M. Wang R. Li S. Shan S. Huang Z. Chen X. Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild Proceedings of the 16th International Conference on multimodal interaction Istanbul, Turkey 12–16 November 2014 494 501
  32. [33]Chen S. Jin Q. Multi-modal conditional attention fusion for dimensional emotion prediction Proceedings of the 24th ACM international conference on Multimedia Amsterdam, The Netherlands 15–19 October 2016 571 575
  33. [34]Choi J.H. Lee J.S. EmbraceNet: A robust deep learning architecture for multimodal classification Inf. Fusion 2019 51 259 270 10.1016/j.inffus.2019.02.010 · DOI
  34. [35]Vaswani A. Shazeer N. Parmar N. Uszkoreit J. Jones L. Gomez A.N. Kaiser L. Polosukhin I. Attention is all you need arXiv 2017 1706.03762
  35. [36]Ho N.H. Yang H.J. Kim S.H. Lee G. Multimodal approach of speech emotion recognition using multi-level multihead fusion attention-based recurrent neural network IEEE Access 2020 8 61672 61686 10.1109/ACCESS.2020.2984368 · DOI
  36. [37]Huang J. Tao J. Liu B. Lian Z. Niu M. Multimodal transformer fusion for continuous emotion recognition Proceedings of the ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) Barcelona, Spain 4–8 May 2020 3507 3511
  37. [38]Siriwardhana S. Kaluarachchi T. Billinghurst M. Nanayakkara S. Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion IEEE Access 2020 8 176274 176285 10.1109/ACCESS.2020.3026823 · DOI
  38. [39]Lian Z. Liu B. Tao J. CTNet: Conversational transformer network for emotion recognition IEEE/ACM Trans. Audio Speech Lang. Process. 2021 29 985 1000 10.1109/TASLP.2021.3049898 · DOI
  39. [41]Poria S. Hazarika D. Majumder N. Naik G. Cambria E. Mihalcea R. Meld: A multimodal multi-party dataset for emotion recognition in conversations arXiv 2018 1810.02508
  40. [42]Chen S.Y. Hsu C.C. Kuo C.C. Ku L.W. Emotionlines: An emotion corpus of multi-party conversations arXiv 2018 1802.08379
  41. [44]Pan S.J. Yang Q. A survey on transfer learning IEEE Trans. Knowl. Data Eng. 2009 22 1345 1359 10.1109/TKDE.2009.191 · DOI · Paper page
  42. [47]Wolf T. Sanh V. Chaumond J. Delangue C. Transfertransfo: A transfer learning approach for neural network based conversational agents arXiv 2019 1901.08149
  43. [48]Kalchbrenner N. Elsen E. Simonyan K. Noury S. Casagrande N. Lockhart E. Stimberg F. Oord A. Dieleman S. Kavukcuoglu K. Efficient neural audio synthesis. In International Conference on Machine Learning Proceedings of the International Conference on Machine Learning Stockholm, Sweden 10 – 15 July 2018
  44. [50]Ouyang X. Kawaai S. Goh E.G.H. Shen S. Ding W. Ming H. Huang D.Y. Audio-visual emotion recognition using deep transfer learning and multiple temporal models Proceedings of the 19th ACM International Conference on Multimodal Interaction Glasgow, UK 13–17 November 2017 577 582
  45. [51]Abdulsalam W.H. Alhamdani R.S. Abdullah M.N. Facial emotion recognition from videos using deep convolutional neural networks Int. J. Mach. Learn. Comput. 2019 9 14 19 10.18178/ijmlc.2019.9.1.759 · DOI
  46. [52]Leong F.H. Deep learning of facial embeddings and facial landmark points for the detection of academic emotions Proceedings of the 5th International Conference on Information and Education Innovations London, UK 26–28 July 2020 111 116
  47. [54]Zhang K. Zhang Z. Li Z. Qiao Y. Joint face detection and alignment using multitask cascaded convolutional networks IEEE Signal Process. Lett. 2016 23 1499 1503 10.1109/LSP.2016.2603342 · DOI · Paper page
  48. [56]Yi D. Lei Z. Liao S. Li S.Z. Learning face representation from scratch arXiv 2014 1411.7923
  49. [58]Hochreiter S. Schmidhuber J. Long short-term memory Neural Comput. 1997 9 1735 1780 10.1162/neco.1997.9.8.1735 9377276 · DOI · Paper page
  50. [60]Franzoni V. Vallverdù J. Milani A. Errors, biases and overconfidence in artificial emotional modeling Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence-Companion Volume Thessaloniki, Greece 14–17 October 2019 86 90

Cited by

Sorted by citations

Authors
Xie, B., Sidulova, M., & Park, C. H.
Affiliations
George Washington University (US)
Corresponding author
Park, Chung Hyuk
ORCID
Venue
Sensors
Publisher
MDPI AG
Published
2021-07-19
Volume
21
Issue
14
Pages
4913
ISSN
1424-8220 (ISSN-L)
DOI
10.3390/s21144913
PMC
PMC8309929
OpenAlex
W3183891373
Cited by
108
References
6
Citations by year
2026: 18 · 2025: 24 · 2024: 19 · 2023: 33 · 2022: 10 · 2021: 4
FWCI
15.95
Type
journal article
Language
en
Open access
Licence
Locations
  • Sensors (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Science Foundation (1846658)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
multimodal emotion recognition, conversation, transformer models, audio, video, text, MELD dataset, robustness, fine-tuning
MeSH terms
Communication, Emotions, Recognition, Psychology, Physical Therapy Modalities

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review