CoChat
Sign Up
Cited by 211Emotion and Mood RecognitionFace recognition and analysisFace and Expression Recognition

Context Based Emotion Recognition Using EMOTIC Dataset

Open access
Kosti, Ronak, Alvarez, Jose, Recasens, Adria, Lapedriza, Agata·2019·IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI 10.1109/tpami.2019.2916866
Open this paper in CoChat — free accountOpen PDF

Abstract

In our everyday lives and social interactions we often try to perceive the emotional states of people. There has been a lot of research in providing machines with a similar capacity of recognizing emotions. From a computer vision perspective, most of the previous efforts have been focusing in analyzing the facial expressions and, in some cases, also the body pose. Some of these methods work remarkably well in specific settings. However, their performance is limited in natural, unconstrained environments. Psychological studies show that the scene context, in addition to facial expression and body pose, provides important information to our perception of people's emotions. However, the processing of the context for automatic emotion recognition has not been explored in depth, partly due to the lack of proper data. In this paper we present EMOTIC, a dataset of images of people in a diverse set of natural situations, annotated with their apparent emotion. The EMOTIC dataset combines two different types of emotion representation: (1) a set of 26 discrete categories, and (2) the continuous dimensions Valence, Arousal, and Dominance. We also present a detailed statistical and algorithmic analysis of the dataset along with annotators' agreement analysis. Using the EMOTIC dataset we train different CNN models for emotion recognition, combining the information of the bounding box containing the person with the contextual information extracted from the scene. Our results show how scene context provides important information to automatically recognize emotional states and motivate further research in this direction.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2019.2916866

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Emotic dataset contains images with the face, upper body, and background.

    “The Emotic [8] dataset, on the other hand, contains images with the face, upper body, and background.”

    From EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge (2023) · cites this paper as [8]

    Section 3 (p. 3) describes EMOTIC as images of people in diverse settings, and Section 4 (p. 8) says the model uses the visible body, including face and head cues, alongside the whole image for scene context. The paper does not specify that the images consistently show the upper body, so that detail is not supported. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Some recent works showed improved performance by considering background contextual information.

    “Consequently, some of recent works showed improved performance by considering background contextual information [5], [8], [9].”

    From EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge (2023) · cites this paper as [8]

  • Contextual information from the scene background helps to make more accurate recognition.

    “Contextual information from the scene background helps to make more accurate recognition.”

    From EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge (2023) · cites this paper as [8]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 40 · Cited by 211

Research works printed but not linked: 36

  1. [1][1] D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang, 'Large-scale visual sentiment ontology and detectors using adjective noun pairs,' in Proceedings of the 21st ACM international conference on Multimedia . ACM, 2013, pp. 223-232. -
  2. [2][2] H. Aviezer, R. R. Hassin, J. Ryan, C. Grady, J. Susskind, A. Anderson, M. Moscovitch, and S. Bentin, 'Angry, disgusted, or afraid? studies on the malleability of emotion perception,' Psychological science , vol. 19, no. 7, pp. 724-732, 2008. -
  3. [3][3] R. Righart and B. De Gelder, 'Rapid influence of emotional scenes on encoding of facial expressions: an erp study,' Social cognitive and affective neuroscience , vol. 3, no. 3, pp. 270-278, 2008. -
  4. [4][4] T. Masuda, P. C. Ellsworth, B. Mesquita, J. Leu, S. Tanida, and E. Van de Veerdonk, 'Placing the face in context: cultural differences in the perception of facial emotion.' Journal of personality and social psychology , vol. 94, no. 3, p. 365, 2008. -
  5. [5][5] L. F. Barrett, B. Mesquita, and M. Gendron, 'Context in emotion perception,' Current Directions in Psychological Science , vol. 20, no. 5, pp. 286-290, 2011. -
  6. [8][8] R. Kosti, J. M. Alvarez, A. Recasens, and A. Lapedriza, 'Emotion recognition in context,' in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017. -
  7. [9][9] M. Pantic and L. J. Rothkrantz, 'Expert system for automatic analysis of facial expressions,' Image and Vision Computing , vol. 18, no. 11, pp. 881-905, 2000. -
  8. [10][10] Z. Li, J.-i. Imai, and M. Kaneko, 'Facial-component-based bag of words and phog descriptor for facial expression recognition.' in SMC , 2009, pp. 1353-1358. -
  9. [12][12] P. Ekman and W. V. Friesen, 'Constants across cultures in the face and emotion.' Journal of personality and social psychology , vol. 17, no. 2, p. 124, 1971. -
  10. [13][13] C. F. Benitez-Quiroz, R. Srinivasan, and A. M. Martinez, 'Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild,' in Proceedings of IEEE International Conference on Computer Vision & Pattern Recognition (CVPR16), Las Vegas, NV, USA , 2016. -
  11. [14][14] M. Soleymani, S. Asghari-Esfeden, Y. Fu, and M. Pantic, 'Analysis of eeg signals and facial expressions for continuous emotion detection,' IEEE Transactions on Affective Computing , vol. 7, no. 1, pp. 17-28, 2016. -
  12. [15][15] S. Du, Y. Tao, and A. M. Martinez, 'Compound facial expressions of emotion,' Proceedings of the National Academy of Sciences , vol. 111, no. 15, pp. E1454-E1462, 2014. -
  13. [16][16] M. A. Nicolaou, H. Gunes, and M. Pantic, 'Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space,' IEEE Transactions on Affective Computing , vol. 2, no. 2, pp. 92-105, 2011. -
  14. [17][17] K. Schindler, L. Van Gool, and B. de Gelder, 'Recognizing emotions expressed by body pose: A biologically inspired neural model,' Neural networks , vol. 21, no. 9, pp. 1238-1246, 2008. -
  15. [18][18] W. Mou, O. Celiktutan, and H. Gunes, 'Group-level arousal and valence recognition in static images: Face, body and context,' in Automatic Face and Gesture Recognition (FG), 2015 11th IEEE International Conference and Workshops on , vol. 5. IEEE, 2015, pp. 1-6. -
  16. [20][20] 'ICML face expression recognition dataset,' https://goo.gl/nn9w4R, accessed: 2017-04-12. -
  17. [21][21] J. L. Tracy, R. W. Robins, and R. A. Schriber, 'Development of a facs-verified set of basic and self-conscious emotion expressions.' Emotion , vol. 9, no. 4, p. 554, 2009. -
  18. [22][22] A. Kleinsmith and N. Bianchi-Berthouze, 'Recognizing affective dimensions from body posture,' in Proceedings of the 2Nd International Conference on Affective Computing and Intelligent Interaction , ser. ACII '07. Berlin, Heidelberg: Springer-Verlag, 2007, pp. 48-58. [Online]. Available: http://dx.doi.org/10.1007/978-3-54074889-2 5 - · DOI
  19. [23][23] A. Kleinsmith, N. Bianchi-Berthouze, and A. Steed, 'Automatic recognition of non-acted affective postures,' IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) , vol. 41, no. 4, pp. 1027-1038, Aug 2011. -
  20. [24][24] T. B¨ anziger, H. Pirker, and K. Scherer, 'Gemep-geneva multimodal emotion portrayals: A corpus for the study of multimodal emotional expressions,' in Proceedings of LREC , vol. 6, 2006, pp. 15-019. -
  21. [25][25] S. Escalera, X. Bar´ o, H. J. Escalante, and I. Guyon, 'Chalearn looking at people: Events and resources,' CoRR , vol. abs/1701.02664, 2017. [Online]. Available: http://arxiv.org/abs/1701.02664 - · arXiv
  22. [26][26] A. Dhall, R. Goecke, J. Joshi, J. Hoey, and T. Gedeon, 'Emotiw 2016: Video and group-level emotion recognition challenges,' in Proceedings of the 18th ACM International Conference on Multimodal Interaction , ser. ICMI 2016. New York, NY, USA: ACM, 2016, pp. 427-432. [Online]. Available: http://doi.acm.org/10.1145/2993148.2997638 - · DOI
  23. [28][28] A. Dhall, J. Joshi, I. Radwan, and R. Goecke, 'Finding happiest moments in a social context,' in Asian Conference on Computer Vision . Springer, 2012, pp. 613-626. -
  24. [29][29] G. Patterson and J. Hays, 'Coco attributes: Attributes for people, animals, and objects,' in European Conference on Computer Vision . Springer, 2016, pp. 85-100. -
  25. [30][30] T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick, 'Microsoft COCO: common objects in context,' CoRR , vol. abs/1405.0312, 2014. [Online]. Available: http://arxiv.org/abs/1405.0312 - · arXiv
  26. [31][31] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, 'Semantic understanding of scenes through ade20k dataset,' 2016. [Online]. Available: https://arxiv.org/pdf/1608.05442 - · arXiv
  27. [32][32] 'Oxford english dictionary,' http://http://www.oed.com, accessed: 2017-06-09. -
  28. [36][36] Y. Groen, A. B. M. Fuermaier, A. E. Den Heijer, O. Tucha, and M. Althaus, 'The empathy and systemizing quotient: The psychometric properties of the dutch version and a review of the cross-cultural stability,' Journal of Autism and Developmental Disorders , vol. 45, no. 9, pp. 2848-2864, 2015. [Online]. Available: http://dx.doi.org/10.1007/s10803-015-2448-z - · DOI
  29. [37][37] B. Zhou, A. Khosla, A. Lapedriza, A. Torralba, and A. Oliva, 'Places: An image database for deep scene understanding,' CoRR , vol. abs/1610.02055, 2015. - · arXiv
  30. [38][38] D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang, 'Large-scale visual sentiment ontology and detectors using adjective noun pairs,' in Proceedings of the 21st ACM international conference on Multimedia . ACM, 2013, pp. 223-232. -
  31. [39][39] T. Chen, D. Borth, T. Darrell, and S.-F. Chang, 'Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks,' arXiv preprint arXiv:1410.8586 , 2014. - · arXiv
  32. [41][41] J. Alvarez and L. Petersson, 'Decomposeme: Simplifying convnets for end-to-end learning,' CoRR , vol. abs/1606.05426, 2016. - · arXiv
  33. [42][42] S. Ioffe and C. Szegedy, 'Batch normalization: Accelerating deep network training by reducing internal covariate shift,' in International Conference on Machine Learning , 2015, pp. 448-456. -
  34. [43][43] R. Caruana, A Dozen Tricks with Multitask Learning , 2012, pp. 163189. -
  35. [44][44] R. Girshick, 'Fast r-cnn,' in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1440-1448. -
  36. [45][45] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database.' in CVPR , 2009. <!-- image --> Ronak Kosti is pursuing his PhD at Universitat Oberta de Catalunya, Spain advised by Prof. Agata Lapedriza. He is working with Scene Understanding and Artificial Intelligence (SUNAI) group in computer vision, specifically in the area of affective computing. He obtained his Masters in Machine Intelligence from DA-IICT (Dhirubhai Ambani Institute of Information and Communication Technology) in 2014. His master's research was based on depth estimation from single image using Artificial Neural Networks. <!-- image --> Jose M. Alvarez is a senior research scientist at NVIDIA. Previously, he was senior deep learning researcher at Toyota Research Institute, US; prior to that he was with Data61, CSIRO, Australia (formerly NICTA) as a researcher. He obtained his Ph.D. in 2010 from Autonomous University of Barcelona under the supervision of Prof. Antonio Lopez and Prof. Theo Gevers. Previous to CSIRO, he worked as a postdoctoral researcher at the Courant Institute of Mathematical Science at New York University under the supervision of Prof. Yann LeCun. <!-- image --> Adria Recasens is pursuing his PhD in computer vision at the Computer Science and Artificial Intelligence Laboratory (CSAIL) of the Massachusetts Institute of Technology advised by Professor Antonio Torralba. His research interests range on various topics in computer vision and machine learning. He is focusing most of his research on automatic gaze-following. He received a Telecommunications Engineer's Degree and a Mathematics Licentiate Degree from the Universitat Politcnica de Catalunya. <!-- image --> Agata Lapedriza is an Associate Professor at the Universitat Oberta de Catalunya. She received her MS deegree in Mathematics at the Universitat de Barcelona and her Ph.D. degree in Computer Science at the Computer Vision Center, at the Universitat Autonoma Barcelona. She was working as a visiting researcher in the Computer Science and Artificial Intelligence Lab, at the Massachusetts Institute of Technology (MIT), from 2012 until 2015. Currently she is a visiting researcher at the MIT Medialab, Affective Computing group from September 2017. Her research interests are related to image understanding, scene recognition and characterization, and affective computing.

Authors
Kosti, R., Alvarez, J., Recasens, A., & Lapedriza, A.
Affiliations
Universitat Oberta de Catalunya (ES)
ORCID
Kosti, Ronak
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2019
Volume
42
Issue
11
Pages
1-1
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
DOI
10.1109/tpami.2019.2916866
OpenAlex
W2946410230
Cited by
211
References
40
Citations by year
2026: 11 · 2025: 39 · 2024: 48 · 2023: 47 · 2022: 29 · 2021: 31 · 2020: 6
FWCI
19.44
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (acceptedVersion)
  • arXiv (Cornell University)
Funders
  • Ministerio de Economia Industria y Competitividad Spain (TIN2015-66951-C2-2-R and RTI2018-095232-B-C22)
  • Innovation and Universities FEDER funds
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
emotion recognition, scene context, valence-arousal-dominance, emotion annotation, annotation agreement, convolutional neural networks, naturalistic images, unconstrained environments

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review