CoChat
Sign Up
Cited by 141Emotion and Mood RecognitionSentiment Analysis and Opinion MiningMultimodal Machine Learning Applications

Survey on AI-Based Multimodal Methods for Emotion Detection

Open access
Marechal, Catherine, Mikołajewski, Dariusz, Tyburek, Krzysztof, Prokopowicz, Piotr, Bougueroua, Lamine, Ancourt, Corinne, Węgrzyn-Wolska, Katarzyna·2019·Lecture Notes in Computer Science
DOI 10.1007/978-3-030-16272-6_11
Open this paper in CoChat — free accountOpen PDF

Abstract

Automatic emotion recognition constitutes one of the great challenges providing new tools for more objective and quicker diagnosis, communication and research. Quick and accurate emotion recognition may increase possibilities of computers, robots, and integrated environments to recognize human emotions, and response accordingly to them a social rules. The purpose of this paper is to investigate the possibility of automated emotion representation, recognition and prediction its state-of-the-art and main directions for further research. We focus on the impact of emotion analysis and state of the arts of multimodal emotion detection. We present existing works, possibilities and existing methods to analyze emotion in text, sound, image, video and physiological signals. We also emphasize the most important features for all available emotion recognition modes. Finally, we present the available platform and outlines the existing projects, which deal with multimodal emotion analysis.

AI snapshot

Written from the paper’s full text.

TL;DR

This survey reviews AI-based multimodal emotion detection across text, sound, image/video, and physiological/motor signals, summarizing methods, datasets, tools, and projects. It reports that automated emotion analysis is feasible but still limited by context, language, real-world noise, and the need for personalization.

Key findings

  • The survey covers emotion analysis in text, sound, image/video, and physiological/motor signals; it highlights multimodal challenges of inter-modality and intra-modality dynamics and cites datasets such as MOSI, ICT-MMMO, and MOUD, plus methods like Tensor Fusion Network and LSTM-based contextual feature extraction.
  • Speech emotion recognition results include a chunk-level classwise averaged recognition rate of almost 70% for the 4-class problem anger, emphatic, neutral, and motherese on the FAU Aibo corpus; Greek speech emotion classification reached up to 75.15%, while Basque reached 80.05% and Spanish 74.82%.
  • Physiological/motor emotion recognition reports include classification rates up to 92% using HRV and PRV fusion (sensitivity 95%, specificity 83.33%), prior general recognition rates from 57.77% to 85.46%, 84.7% by Jang et al., EEG average overall accuracy of 77.14% with MLPNN and 72.92% with kNN, highest gait accuracy of 80.5%, 54.5% for combining heart rate variability, respiration frequency, and facial expressions, and 61.39% for the best i2b2/VA/Cincinnati challenge team.
  • Related projects and tools include openSMILE, openEAR (capable of generating >500 k features in real time with RTF < 0.1), ASC-Inclusion, INTERSPEECH ComParE, IntraFace, Microsoft Azure Emotion API, and Ekman micro-expression training tools.

Methods

  • Design: a survey/state-of-the-art review of automated emotion representation, recognition, and prediction, with emphasis on multimodal emotion detection.
  • Data or sample: no primary experimental sample; the paper reviews existing works, datasets/corpora, platforms, and projects, including MOSI, ICT-MMMO, MOUD, FAU Aibo, and MAHNOB-HCI.
  • Main analysis: narrative synthesis organized by modality—text, sound, image/video, physiological and motor signals—followed by related projects and future directions.

Limitations

  • Stated: limitations of current knowledge and experience concern tools for automatic non-invasive emotion measurement and analysis; key requirements include portability, non-intrusiveness, and low price, and further research is needed.
  • Stated: speech emotion analysis highly depends on language; real-world factors such as background noise and non-speech vocalizations degrade results from controlled environments, and algorithms must be personalized to each person to be reliable.
  • Evident from the text: as a survey, it provides no new experimental validation or systematic meta-analysis, and the authors note they have addressed only some of the most important issues.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/978-3-030-16272-6_11

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 66 cited sources checked · 0 supported · 0 flagged

  • Premisecitation

    Several works on depression have emerged based on social networks: Twitter and Facebook.

    “Several works on depression have emerged.They are based on social networks: Twitter [23,24] and Facebook [25,26].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [23] Wang, W., Hernandez, I., Newman, D.A., He, J., Bian, J.: Twitter analysis: studying US weekly trends in work stress and emotion. Appl. Psychol. 65(2), 355–378 (2016) source

      Not checked in the free run.

    • cites [24] Reece, A.G., Reagan, A.J., Lix, K.L., Dodds, P.S., Danforth, C.M., Langer, E.J.: Forecasting the onset and course of mental illness with Twitter data (Unpublished manuscript). https://arxiv.org/pdf/1608.07740.pdf

      Not checked in the free run.

    • cites [25] Park, J., Lee, D.S., Shablack, H., et al.: When perceptions defy reality: the relationships between depression and actual and perceived Facebook social support. J. Affect. Disord. 200, 37–44 (2016) source

      Not checked in the free run.

    • cites [26] Burke, M., Develin, M.: Once more with feeling: supportive responses to social sharing on Facebook. In: Proceedings of the ACM 2016 Conference on Computer Supported Cooperative Work, pp. 1462–1474 (2016) source

      Not checked in the free run.

  • Premisecitation

    Several multimodal datasets include sentiment annotations.

    “Several multimodal datasets include sentiment annotations.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [6] Zadeh, A., Zellers, R., Pincus, E., Morency, L.P.: Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intell. Syst. 31(6), 82–88 (2016). https://doi.org/10.1109/mis.2016.94 source

      Not checked in the free run.

    • cites [7] Wöllmer, M., et al.: YouTube movie reviews: sentiment analysis in an audio-visual context. IEEE Intell. Syst. 28(3), 46–53 (2013) source

      Not checked in the free run.

    • cites [8] Perez-Rosas, V., Mihalcea, R., Morency, L.P.: Utterance-level multimodal sentiment analysis. In: ACL, vol. 1, pp. 973–982 (2013)

      Not checked in the free run.

  • Premisecitation

    Zadeh et al. introduced the first multimodal dataset (MOSI) with opinion-level sentiment intensity annotations.

    “Zadeh et al. introduced the first multimodal dataset (MOSI) with opinion-level sentiment intensity annotations and studying the prototypical interaction patterns between facial gestures and spoken words when inferring sentiment intensity.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] Zadeh, A., Zellers, R., Pincus, E., Morency, L.P.: Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intell. Syst. 31(6), 82–88 (2016). https://doi.org/10.1109/mis.2016.94 source

      Not checked in the free run.

  • Premisecitation

    Zadeh introduced the Tensor Fusion Network (TFN), which combines intra-modality and inter-modality models.

    “To solve this problem, Zadeh introduced the Tensor Fusion Network (TFN), which combine the intra-modality and inter-modality models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L.P.: Tensor fusion network for multimodal sentiment analysis, arXiv:1707.07250 . In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 7–11 September 2017, Copenhagen, Denmark, pp. 1103–1114. Association for Computational Linguistics source

      Not checked in the free run.

  • Premisecitation

    Zadeh et al. studied prototypical interaction patterns between facial gestures and spoken words when inferring sentiment intensity.

    “Zadeh et al. introduced the first multimodal dataset (MOSI) with opinion-level sentiment intensity annotations and studying the prototypical interaction patterns between facial gestures and spoken words when inferring sentiment intensity.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] Zadeh, A., Zellers, R., Pincus, E., Morency, L.P.: Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intell. Syst. 31(6), 82–88 (2016). https://doi.org/10.1109/mis.2016.94 source

      Not checked in the free run.

  • Classified as probably not a substantial claim. 55
  • May combine two claims. 21

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Poria et al. developed an LTSM-based network to extract contextual features from video for multimodal sentiment analysis.

    “Interesting work is realized by Poria et al., who developed a LTSM-based network to extract contextual features from the video for multimodal sentiment analysis [10].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10] Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., Morency, L.P.: Context-dependent sentiment analysis in user-generated videos. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 873–883 (2017) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    ICT-MMMO and MOUD are examples of multimodal datasets.

    “For other examples of data sets we can cite ICT-MMMO [7] and MOUD [8] datasets.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [7] Wöllmer, M., et al.: YouTube movie reviews: sentiment analysis in an audio-visual context. IEEE Intell. Syst. 28(3), 46–53 (2013) source

      Not checked in the free run.

    • cites [8] Perez-Rosas, V., Mihalcea, R., Morency, L.P.: Utterance-level multimodal sentiment analysis. In: ACL, vol. 1, pp. 973–982 (2013)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisereasoning

    It is not yet possible to find an ideal case for marking opinion in a text written by different users, because the text does not follow the rules.

    “It is not yet possible to find out an ideal case to marking the opinion in a text written by different users, because the text does not follow the rules.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    With an average of 500 million messages sent per day, Twitter seems ideal for tracking opinions on various subjects.

    “With an average of 500 million messages sent per day, the platform seems ideal for tracking opinions on various subjects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    It is impossible to schedule every possible case in sentiment analysis.

    “Therefore, it is impossible to schedule every possible case.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Very often the same phrase can be considered positive for one person and negative for another.

    “Moreover, very often the same phrase can be considered as positive for one person and negative for another one.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Automated interpretation of Twitter messages is complicated by embedded links, abbreviations and misspellings.

    “However, automated interpretation is complicated by embedded links, abbreviations and misspellings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    On the Internet, everyone uses his or her own style and vocabulary, adding extra difficulty to sentiment analysis.

    “In addition, on the Internet, everyone uses his or her own style and vocabulary, what adds extra difficulty to the task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Premisenone given

    In the current growth of artificial intelligence, machine learning and natural language processing, driven by new technological possibilities, it is possible to automate the analysis of vast amounts of publicly published data.

    “In the current growth of artificial intelligence, machine learning and natural language processing, driven by new technological possibilities, it is possible to automate the analysis of vast amounts of publicly published data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Several authors have been interested in the use of emoticons to complete sentiment analysis.

    “Several authors have been interested in the use of emoticons to complete the sentiment analysis.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [27] Go, A., Bhayani, R., Huang, L.: Twitter sentiment classification using distant supervision. J. CS224N Proj. Rep., Stanford 1, 12 (2009)

      Not checked in the free run.

    • cites [28] Liu, K.L., Li, W.J., Guo, M.: Emoticon smoothed language models for Twitter sentiment analysis. In: AAAI (2012)

      Not checked in the free run.

    • cites [29] Węgrzyn-Wolska, K., Bougueroua, L., Yu, H., Zhong, J.: Explore the effects of emoticons on Twitter sentiment analysis. In: Proceedings of Third International Conference on Computer Science & Engineering (CSEN 2016), 27–28 August 2016, Dubai, UAE source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Opinions and sentiments are highly sensitive to context and field; the same string might be positive in one context and negative in another.

    “Moreover, opinions and sentiments are highly sensitive to the context and the field in which they are used: the same string might be positive in one context and negative in another.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Zadeh et al. proposed a new computational representation called multimodal dictionary based on a language-gesture study and evaluated it in a speaker-independent model for sentiment intensity prediction.

    “They proposed a new computational representation, called multimodal dictionary, based on a language-gesture study and evaluated the proposed approach in a speaker-independent model for sentiment intensity prediction [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] Zadeh, A., Zellers, R., Pincus, E., Morency, L.P.: Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intell. Syst. 31(6), 82–88 (2016). https://doi.org/10.1109/mis.2016.94 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Poria et al. presented a multimodal sentiment analysis framework including sets of relevant features for text and visual data and a simple technique for fusing features extracted from different modalities.

    “Additionally, they presented a multimodal sentiment analysis framework, which includes sets of relevant features for text and visual data, as well as a simple technique for fusing the features extracted from different modalities [11].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [11] Poria, S., Cambria, E., Howard, N., Huang, G.B., Hussain, A.: Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing 174(Part A), 50–59 (2016). https://doi.org/10.1016/j.neucom.2015.01.095 . ISSN 0925-2312 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Contextual changes of polarity indicating words, such as negation, sarcasm as well as weak syntactic structures make it troublesome for both machines and humans to safely determine polarity of messages.

    “Contextual changes of polarity indicating words, such as negation, sarcasm as well as weak syntactic structures make it troublesome for both machines and humans to safely determine polarity of messages.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The automation of sentiment detection on social networks has gained attention for various purposes.

    “The automation of sentiment detection on these social networks has gained attention for various purposes [18][19][20].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [18] Ghiassi, M., Skinner, J., Zimbra, D.: Twitter brand sentiment analysis: a hybrid system using n-gram analysis and dynamic artificial neural network. Expert Syst. Appl. 40(16), 6266–6282 (2013) source

      Not checked in the free run.

    • cites [19] Zhou, X., Tao, X., Yong, J., Yang, Z.: Sentiment analysis on tweets for social events. In: Proceedings of the 2013 IEEE 17th International Conference on Computer Supported Cooperative Work in Design, CSCWD 2013, 27–29 June 2013, pp. 557–562 (2013) source

      Not checked in the free run.

    • cites [20] Salathé, M., Vu, D.Q., Khandelwal, S., Hunter, D.R.: The dynamics of health behavior sentiments on a large online social network. EPJ Data Sci. 2, 4 (2013). https://doi.org/10.1140/epjds16 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Intra-modality dynamics is modeled through three Modality Embedding Subnetworks for language, visual and acoustic modalities.

    “Intra-modality dynamics is modeled through three Modality Embedding Subnetworks, for language, visual and acoustic modalities, respectively [9].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L.P.: Tensor fusion network for multimodal sentiment analysis, arXiv:1707.07250 . In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 7–11 September 2017, Copenhagen, Denmark, pp. 1103–1114. Association for Computational Linguistics source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    According to the Kepios study, the number of active users of social networks increased by 13% in 2017 to reach 3.3 billion users in April 2018.

    “According to the Kepios study [17], the number of active users of social networks increased by 13% in 2017 to reach 3.3 billion users in April 2018.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] Kepios: Digital in 2018, essential insights into internet, social media, mobile, and ecommerce use around the world, April 2018. https://kepios.com/data/

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The AutoEmotive (MIT Media Lab) is a prototype equipped with sensors and a camera on the steering wheel that measures driver stress and fatigue and, when needed, plays background music, changes temperature and light, or proposes a less stressful journey.

    “The AutoEmotive (MIT Media Lab) is a prototype equipped with sensors and a camera placed on the steering wheel [5].This vehicle measures the level of stress and fatigue of the driver.When the need arises, he puts a background music, changes the temperature and light in the vehicle interior, or still proposes to follow a less stressful journey.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] Hernandez, J., et al.: AutoEmotive: bringing empathy to the driving experience to manage stress. In: DIS 2014, 21–25 June 2014, Vancouver, BC, Canada. ACM (2014). http://dx.doi.org/10.1145/2598784.2602780 . 978-1-4503-2903-3/14/06 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The first study in the field of emotion detection was born during the sixties/seventies, with mood rings as the most prominent example.

    “The first study in the field of emotion detection was born during the sixties/seventies.The most prominent example is that of mood rings [2].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2] Mood Ring Monitors Your State of Mind, Chicago Tribune, 8 October 1975, at C1: Ring Buyers Warm Up to Quartz Jewelry That Is Said to Reflect Their Emotions. The Wall Street Journal, 14 October 1975, at p. 16; and “A Ring Around the Mood Market”, The Washington Post, 24 November 1975, at B9

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The frequent expression of negative emotion words on social media has been linked to depression.

    “The frequent expression of negative emotion words on social media has been linked to depression.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Seabrook, E.M., Kern, M.L., Fulcher, B.D., Rickard, N.S.: Predicting depression from language-based emotion dynamics: longitudinal analysis of Facebook and Twitter status updates. J. Med. Internet Res. 20(5), e168 (2018). https://doi.org/10.2196/jmir.9267 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisereasoning

    The very short format of Twitter messages facilitates classification since short messages rarely discuss more than one topic.

    “Furthermore, the very short format messages facilitate classification since short messages rarely discuss more than one topic.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The limitations of current knowledge and experience still concern tools for automatic non-invasive emotion measurement and analysis.

    “Presented approach may be successful, but the limitations of the current knowledge and experience still concern tools for automatic non-invasive emotion measurement and analysis.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Afzal et al.'s experiments confirmed that human judgement is still the best, with the highest recognition rates, followed by stick-figure models, point-light displays, and then XFace animations.

    “Their experiences confirmed that the human judgement is still the best one, with the highest recognition rates, followed by stick-figure models, point-light displays, and then XFace animations.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48] Afzal, S., Sezgin, T.M., Gao, Y., Robinson, P.: Perception of emotional expressions in different representations using facial feature points. IEEE (2009). 978-1-4244-4799 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The author in [35] focused on mono-modal systems with speech as only input channel.

    “The author in [35] focused on mono-modal systems with speech as only input channel.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] Steidl, S.: Automatic classification of emotion-related user states in spontaneous children’s speech. Ph.D. thesis, Erlangen (2009)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    Nonverbal behavior allows communication of even complex social concepts in various contexts, so it may be regarded not only as supplementary but also as a basic way of emotion recognition.

    “It allows to communicate even complex social concepts, in various contexts, thus may be regarded not only supplementary, but also as basic way of emotion recognition.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The author in [35] proposed FAU Aibo Emotion Corpus, a speech corpus with naturally occurring emotion-related states.

    “He proposed FAU Aibo Emotion Corpus, which is a speech corpus with naturally occurring emotion-related states.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] Steidl, S.: Automatic classification of emotion-related user states in spontaneous children’s speech. Ph.D. thesis, Erlangen (2009)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Facial expressions as a noninvasive method are used in behavioral science and in clinical settings.

    “Thus, facial expressions as a noninvasive method are used in behavioral science and in clinical.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Ekman and Friesen developed in 1978 the Facial Action Coding System (FACS), a special system for objectively measuring facial movement, to reduce the complexity of automatic affective inference, measurement and interpretation of facial expressions.

    “To reduce the complexity of automatic affective inference, measurement and interpretation of facial expressions, Ekman and Friesen developed in 1978 special system for objectively measuring facial movement; the Facial Action Coding System (FACS) [45].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] Ekman, P., Friesen, W., Hager, J.: Facial action coding system: Research Nexus. Network Research Information, Salt Lake City (2002)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    There are three main dimensions of facial variation: morphology, complexion and dynamics.

    “There are three main dimensions of facial variation: morphology, complexion and dynamics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Authors in [27] utilize Twitter API to get training data containing emoticons like :) and :(, using these emoticons as noisy labels; tweets with :) are thought positive and tweets with :( negative.

    “Authors in [27] utilize Twitter API to get training data that contain emoticons like :) and :(.They use these emoticons as noisy labels.Tweets with :) are thought to be positive training data and tweets with :( are thought to be negative training data.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27] Go, A., Bhayani, R., Huang, L.: Twitter sentiment classification using distant supervision. J. CS224N Proj. Rep., Stanford 1, 12 (2009)

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Emotion analysis of speech is possible; however, it highly depends on the language.

    “Emotion analysis of speech is possible; however, it highly depends of the language.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Authors in [29] explored the influence of emoticons on Twitter sentiment analysis.

    “The authors in [29] explored the influence of emoticons on TSA.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29] Węgrzyn-Wolska, K., Bougueroua, L., Yu, H., Zhong, J.: Explore the effects of emoticons on Twitter sentiment analysis. In: Proceedings of Third International Conference on Computer Science & Engineering (CSEN 2016), 27–28 August 2016, Dubai, UAE source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Eleven emotion-related states are labeled on the word level in FAU Aibo.

    “Eleven emotion-related states are labeled on the word level.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] Steidl, S.: Automatic classification of emotion-related user states in spontaneous children’s speech. Ph.D. thesis, Erlangen (2009)

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    Morphology and complexion deal with static dimensions and are useful for facial recognition, but their role for emotion analysis is not significant.

    “As first two dimensions, morphology and complexion, deals with static dimensions, there are very useful for facial recognition but their role for emotion analyze is not significant.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    In the real world, factors such as background noise and non-speech voice like laugh, whimper, cry, sigh, etc., greatly aggravate the results obtained in a controlled environment.

    “There are many factors such as background noise and not speech voice like a laugh, a whimper, a cry, a sigh, etc., which greatly aggravate the results obtained in a controlled environment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Not only basic emotions can be analyzed, but also the cognitive assessment and their physiological response.

    “Not only basic emotions can be analyzed, but also the cognitive assessment (interpretation of stimuli in the environment) and their physiological response.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    The face is equipped with a large number of independent muscles, which can be activated at different levels of intensity.

    “The face is equipped with a large number of independent muscles, which can be activated at different levels of intensity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The three automatically generated representations were point-based display, stick figure models, and 3D XFace Animation.

    “First, the point-based display was created from the output of a commercial automatic face-tracker [49] on a black background.Second one, stick figure models, which connecting the automatically tracked feature-points using straight lines and sketching eyes using typical shape.The last one, 3D XFace Animation was a 3D animated facial expressions and displays created using XFace open source toolkit.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [48] Afzal, S., Sezgin, T.M., Gao, Y., Robinson, P.: Perception of emotional expressions in different representations using facial feature points. IEEE (2009). 978-1-4244-4799 source

      Not checked in the free run.

    • cites [49] http://www.nevenvision.com . Licensed from Google Inc.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recognition of emotions based on the voice has been studied for decades.

    “The recognition of emotions based on the voice has been studied for decades [31][32][33][34].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31][32][33][34]

      Not checked in the free run.

  • May combine two claims.

    Interpretationnone given

    Automated emotion analysis is possible and can be very useful for improving the exactness of computer/machine reaction and making possible to anticipate the emotional state of the interlocutor more quickly.

    “We showed that automated emotion analysis is possible and can be very useful for improving the exactness of the computer/machine reaction and making possible to anticipate the emotional state of the interlocutor more quickly.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    FACS, based on a system originally developed by Hjortsjö, became the standard for identifying any movement of the face.

    “FACS, based on a system originally developed by a Swedish anatomist named Hjortsjö [46] became the standard for identifying any movement of the face.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [45] Ekman, P., Friesen, W., Hager, J.: Facial action coding system: Research Nexus. Network Research Information, Salt Lake City (2002)

      Not checked in the free run.

    • cites [46] Hjorztsjö, C.H.: Man’s face and mimic language (1969). https://books.google.com/books/about/Man_s_Face_and_Mimic_Laguage.html?id=BakQAQAAIAAJ

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Later, Ekman and Sejnowski studied computer based facial measurements.

    “Later, Ekman and Sejnowski studied also computer based facial measurements [47].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [47] Ekman, P., Huang, T.S., Sejnowski, T.J., et al.: Final report to NSF of the planning workshop on facial expression understanding, vol. 378. Human Interaction Laboratory, University of California, San Francisco (1993)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    The eyes and mouth have high importance to emotion detection; to successfully recognize an emotion, observations mostly rely on the eye and mouth regions.

    “The eyes and mouth have high importance to emotion detection; therefore, to successfully recognize an emotion, the observations mostly rely on the eye and mouth regions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisereasoning

    The actions of eyes and mouth allowed grouping the expressions in a continuous space, ranging from sadness and fear (reliance on the eyes) to disgust and happiness (mouth).

    “Furthermore, the actions of eyes and mouth allowed grouping the expressions in a continuous space, ranging from sadness and fear (reliance on the eyes) to disgust and happiness (mouth).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Consequently, the face is capable of generating a high number of complex dynamic facial expression patterns.

    “Consequently, the face is capable of generating a high number of complex dynamic facial expression patterns.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    FACS is a common standard to systematically categorize and index the physical expression of emotions.

    “FACS is a common standard to systematically categorize and to index the physical expression of emotions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    FAU Aibo is a corpus of spontaneous, emotionally colored speech of German children aged 10 to 13 interacting with the Sony Robot Aibo.

    “FAU Aibo is a corpus of spontaneous, emotionally colored speech of German children at the age of 10 to 13 years interacting with the Sony Robot Aibo.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] Steidl, S.: Automatic classification of emotion-related user states in spontaneous children’s speech. Ph.D. thesis, Erlangen (2009)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Combining observations of eyes and mouth with facial AUs increases knowledge about the areas involved in displaying each emotion.

    “Combining these observations with facial AUs increase knowledge about the areas involved in displaying each emotion.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In [28], authors present ESLAM (Emoticon Smoothed LAnguage Models) which combine fully supervised methods and distantly supervised methods.

    “In [28], authors present the ESLAM (Emoticon Smoothed LAnguage Models) which combine fully supervised methods and distantly supervised methods.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] Liu, K.L., Li, W.J., Guo, M.: Emoticon smoothed language models for Twitter sentiment analysis. In: AAAI (2012)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Automatic emotion analysis requires advanced recognition and modeling, very often based on artificial intelligence systems.

    “Automatic emotion analysis requires advanced recognition and modeling, very often based on artificial intelligence systems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    These six characteristics are common to all humans, confirming Darwin's evolutionary thesis.

    “They are common to all humans confirming Darwin's evolutionary thesis.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [43] Ekman, P.: Facial expression and emotion. Am. Psychol. 48, 384–392 (1993) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The majority of voice emotion recognition systems are speech-oriented.

    “In addition, the majority of such a system are speech-oriented.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Nonverbal behavior constitutes a useful means of communication in addition to spoken language.

    “Nonverbal behavior constitutes useful means of communication in addition to spoken language.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    It is possible to register facial information automatically in real-time, without requiring any specialized equipment except a simple video camera.

    “It is possible to register the facial information automatically in real-time, without requiring any specialized equipment except a simple video camera.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent research has shown that under certain circumstances multimodal emotion recognition is possible even in real time.

    “Recent research has shown that under certain circumstances multimodal emotion recognition is possible even in real time [37].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37] Kukolja, D., Popović, S., Horvat, M., Kovač, B., Ćosić, K.: Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications. Int. J. Hum.-Comput. Stud. 72(10), 717–727 (2014) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Voice-based emotion recognition systems relying on audio input have low requirements for hardware.

    “Voice-based emotion recognition system relying on audio input has low requirements for hardware.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Human gait reflects the walker's emotional state.

    “They showed that human gait reflects the walker's emotional state.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [62] Li, S., Cui, L., Zhu, C., Li, B., Zhao, N., Zhu, T.: Emotion recognition using Kinect motion capture data of human gaits. PeerJ 4, e2364 (2016) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Such an approach may cause quicker development of more user-friendly systems and environments supporting everyday life and work.

    “Such an approach may cause quicker development of more user-friendly systems and environments supporting everyday life and work.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    A larger set of analyzed signals and more recognized emotions provides better results.

    “that more higher set of analyzed signals and more recognized emotions provides better results [64][65][66].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [64] Khezri, M., Firoozabadi, M., Sharafat, A.R.: Reliable emotion recognition system based on dynamic adaptive fusion of forehead biopotentials and physiological signals. Comput. Methods Programs Biomed. 122(2), 149–164 (2015) source

      Not checked in the free run.

    • cites [65] Gouizi, K., Bereksi Reguig, F., Maaoui, C.: Emotion recognition from physiological signals. J. Med. Eng. Technol. 35(6–7), 300–307 (2011) source

      Not checked in the free run.

    • cites [66] Verma, G.K., Tiwary, U.S.: Multimodal fusion framework: a multiresolution approach for emotion classification and recognition from physiological signals. Neuroimage 102(Part 1), 162–172 (2014) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The best of 24 participants' teams in the 2011 i2b2/VA/Cincinnati Medical Natural Language Processing Challenge achieved accuracy of 61.39%.

    “Results of the 2011 i2b2/VA/Cincinnati Medical Natural Language Processing Challenge showed that the best of 24 participants' teams achieved accuracy of 61.39% [67].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [67] Yang, H., Willis, A., de Roeck, A., Nuseibeh, B.: A hybrid model for automatic emotion recognition in suicide notes. Biomed. Inform. Insights 5(Suppl. 1), 17–30 (2012)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    Novel intelligent systems may be friendlier, preventing the computers from acting inappropriately.

    “Novel intelligent systems may be friendlier, preventing the computers from acting inappropriately.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Several popular commercial packages offer specific facial image analysis tasks, including facial expression recognition, facial attribute analysis, and face tracking.

    “Several popular commercial packages offer specific facial image analysis tasks, including facial expression recognition, facial attribute analysis, and face tracking.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Özerdem and Polat achieved average overall accuracy of 77.14% using MLPNN and 72.92% using kNN on EEG signals.

    “Özerdem and Polat used EEG signal -resultant average overall accuracy was 77.14% (using MLPNN) and 72.92% (using kNN) [56].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [56] Wei, W., Jia, Q., Feng, Y., Chen, G.: Emotion recognition based on weighted fusion strategy of multichannel physiological signals. Comput. Intell. Neurosci. 2018, 9 (2018). 5296523 source

      Not checked in the free run.

63 of 63

  • The paper relies on this without support.

    One of the most challenges in multimodal emotion analysis is to model the interactions between language, visual and acoustic behaviors that change the observation of the expressed emotion (inter-modality dynamics).

    “One of the most challenges in multimodal emotion analysis is to model the interactions between language, visual and acoustic behaviors that change the observation of the expressed emotion (named the inter-modality dynamics).”

  • The paper relies on this without support.

    A second challenge in multimodal emotion analysis (intra-modality dynamics) is to efficiently explore emotion on highly expressive nature modality.

    “A second challenge in multimodal emotion analysis (named intra-modality dynamics) is to efficiently explore emotion, not only on one but on highly expressive nature modality (ex.spoken language where proper language structure is often ignored, video and acoustic modalities which are expressed through both space and time.”

  • The paper relies on this without support.

    Traditional emotion recognition and classification associate the 'value' to at least several basic emotions such as happiness, joy, neutrality, surprise, sadness, disgust, anger, or fear.

    “Traditional emotion recognition and classification associate the “value” to at least several basic emotions such as happiness, joy, neutrality, surprise, sadness, disgust, anger, or fear.”

  • The paper relies on this without support.

    Multimodal emotion analysis has become a very popular research domain.

    “Multimodal emotion analysis becomes very popular research domain.”

  • The paper relies on this without support.

    The inability of today's systems to recognize, express and feel emotions limits their ability to act intelligently and interact naturally with humans.

    “The inability of today's systems to recognize, express and feel emotions limits their ability to act intelligently and interact naturally with humans.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Marechal et al. (2019) show that affective state can be extracted both from the user's pose, physiological features like blood pressure, and from facial features.

    “In a survey on multimodal approaches for emotion detection, Marechal et al. (2019) show that affective state can be extracted both from the user's pose (Zacharatos et al., 2014), physiological features like blood pressure (Shu et al., 2018), and from facial features (Ekman, 1993).”

    From Modeling Feedback in Interaction With Conversational Agents—A Review (2022) · cites this paper as Marechal et al., 2019

  • [32,33] examined multimodal practices limited to specific use cases, including emotion recognition and human activity/context recognition.

    “Recently, the valuable works of [32, 33] considered several multimodal practices that apply only to specific multimodal use cases and applications, such as emotion recognition [32], human activity and context recognition [33].”

    From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [32]

  • No recent work directly covers the latest advances in deep multimodal learning specifically for the computer vision community.

    “but to the best of our knowledge, there is no recent work that directly addresses the latest advances in deep multimodal learning particularly for the computer vision community.”

    From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [32]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • IntraFace citation points to an unrelated reference

    Minor

    The text cites [50] for IntraFace, but reference 50 is a paper by Amira et al. on monitoring chronic disease. IntraFace is listed as reference 49.

  • Jang citation points to a gait study

    Minor

    The text cites [58] for research by Jang et al., but reference 58 is a gait-based emotion-recognition study by Li et al. Jang et al. are listed at reference 54.

  • Goshvarpour accuracy cites an unrelated paper

    Minor

    The reported classification results are attributed to Goshvarpour et al. with citation [63], but reference 63 is a paper on emotion recognition in suicide notes. The Goshvarpour paper is listed at reference 59.

  • Emotion API citation points to a physiological-signals review

    Minor

    The text identifies [51] as the source for Microsoft's Emotion API, whereas reference 51 is a review of emotion recognition using physiological signals.

  • Amira citation points to a Jang study

    Minor

    The text attributes the cited healthcare work to Amira et al., but reference 54 is by Jang et al. The Amira et al. work appears at reference 50.

  • Kortelainen citation points to Goshvarpour

    Minor

    The text attributes the cited physiological-and-facial-expression study to Kortelainen et al., but reference 59 is by Goshvarpour and Abbasi. The Kortelainen et al. study is listed at reference 55.

  • Physiological-signals review cites a different paper

    Minor

    The described review is cited as [55], but reference 55 is a preliminary study combining physiological signals and facial expressions; the review is listed at reference 51.

  • Decision-level fusion citation points to a body-movement survey

    Minor

    The text cites [56] for multichannel physiological-signal fusion, but reference 56 is a survey of body-movement analysis. The weighted-fusion paper is listed at reference 52.

  • Özerdem and Polat citation points to Tsui et al.

    Minor

    The text cites [57] for Özerdem and Polat's EEG study, but reference 57 is a keystroke study by Tsui et al. The Özerdem and Polat study is listed at reference 53.

  • Medical NLP Challenge result cites a paralinguistics paper

    Minor

    The cited reference [67] is about the computational paralinguistics challenge, not the 2011 medical NLP challenge whose result the text reports.

  • Ekman training-tools citation points to an EEG study

    Minor

    The cited reference [53] is an EEG-based emotion-recognition paper by Özerdem and Polat, not a source by Ekman about training tools.

11 of 11

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Mood Ring Monitors Your State of Mind, Chicago Tribune, 8 October 1975, at C1: Ring Buyers Warm Up to Quartz Jewelry That Is Said to Reflect Their Emotions. The Wall Street Journal, 14 October 1975, at p. 16; and “A Ring Around the Mood Market”, The Washington Post, 24 November 1975, at B9
  2. Dziczkowski, G., Węgrzyn-Wolska, K.: Tool of the intelligence economic: recognition function of reviews critics. In: ICSOFT 2008 Proceedings. INSTICC Press (2008)
  3. Go, A., Bhayani, R., Huang, L.: Twitter sentiment classification using distant supervision. J. CS224N Proj. Rep., Stanford 1, 12 (2009)
  4. Steidl, S.: Automatic classification of emotion-related user states in spontaneous children’s speech. Ph.D. thesis, Erlangen (2009)
  5. Ekman, P., Huang, T.S., Sejnowski, T.J., et al.: Final report to NSF of the planning workshop on facial expression understanding, vol. 378. Human Interaction Laboratory, University of California, San Francisco (1993)

Cited by

Sorted by citations

Related

Authors
Marechal, C., Mikołajewski, D., Tyburek, K., Prokopowicz, P., Bougueroua, L., Ancourt, C., & Węgrzyn-Wolska, K.
Affiliations
  • Kazimierz Wielki University in Bydgoszcz (PL)
  • Engineering School of Information and Digital Technologies (FR)
  • Université Paris Sciences et Lettres (FR)
Corresponding author
Węgrzyn-Wolska, Katarzyna
ORCID
Venue
Lecture Notes in Computer Science
Publisher
Springer International Publishing
Published
2019
Volume
11400
Pages
307-324
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
DOI
10.1007/978-3-030-16272-6_11
OpenAlex
W2922915988
Semantic Scholar
059ce4ee466aa3d85bd351970b7e4716d7c9cfd9
Cited by
137
References
63
Citations by year
2026: 6 · 2025: 11 · 2024: 25 · 2023: 17 · 2022: 24 · 2021: 34 · 2020: 19 · 2019: 1
FWCI
24.56
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion, cc-by)
  • HAL (submittedVersion, other-oa)
Funders
Uniwersytet Kazimierza Wielkiego w Bydgoszczy
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
multimodal emotion recognition, automatic emotion recognition, emotion analysis, text emotion analysis, speech emotion analysis, image-based emotion recognition, video-based emotion recognition, physiological signals, emotion prediction, emotion representation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review