CoChat
Sign Up
Cited by 323Emotion and Mood RecognitionFace recognition and analysisFace and Expression Recognition

Context-Aware Emotion Recognition Networks

Open access
Lee, Jiyoung, Kim, Seungryong, Kim, Sunok, Park, Jungin, Sohn, Kwanghoon·2019·2019 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv.2019.01024
Open this paper in CoChat — free accountOpen PDF

Abstract

Traditional techniques for emotion recognition have focused on the facial expression analysis only, thus providing limited ability to encode context that comprehensively represents the emotional responses. We present deep networks for context-aware emotion recognition, called CAER-Net, that exploit not only human facial expression but also context information in a joint and boosting manner. The key idea is to hide human faces in a visual scene and seek other contexts based on an attention mechanism. Our networks consist of two sub-networks, including two-stream encoding networks to separately extract the features of face and context regions, and adaptive fusion networks to fuse such features in an adaptive fashion. We also introduce a novel benchmark for context-aware emotion recognition, called CAER, that is appropriate than existing benchmarks both qualitatively and quantitatively. On several benchmarks, CAER-Net proves the effect of context for emotion recognition. Our dataset is available at http://caer-dataset.github.io.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv.2019.01024

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The CAER and CAER-S datasets encompass video clips and images from TV shows, primarily focusing on upper body and facial images with background.

    “The CAER and CAER-S [51] datasets encompass video clips and images from TV shows, primarily focusing on upper body and facial images with background.”

    From EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge (2023) · cites this paper as [51]

    Section 4.1 (p. 5) says CAER consists of video clips collected from TV shows and CAER-S is a static-image subset drawn from it. The Introduction (p. 2) describes using facial expressions alongside scene context, but our copy does not establish that the datasets primarily focus on upper bodies. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • The CAER-S dataset is a collection of nearly 70,000 facial expressions from 79 different TV shows, manually annotated into six distinct emotion categories.

    “The CAER-S dataset is a collection of nearly 70,000 facial expressions from 79 different TV shows. These data are manually annotated into six distinct emotion categories.”

    From SAFER: Situation Aware Facial Emotion Recognition (2023) · cites this paper as [34]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Some recent works showed improved performance by considering background contextual information.

    “Consequently, some of recent works showed improved performance by considering background contextual information [5], [8], [9].”

    From EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge (2023) · cites this paper as [9]

  • Contextual information from the scene background helps to make more accurate recognition.

    “Contextual information from the scene background helps to make more accurate recognition.”

    From EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge (2023) · cites this paper as [9]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. [1][1] Sidney D'Mello, Rosalind W Picard, and Arthur Graesser. Toward an affect-sensitive autotutor. IEEE Int. Systems , 2007. -
  2. [3][3] Georgios N Yannakakis and Julian Togelius. Experiencedriven procedural content generation. IEEE Trans. AC , 2011. -
  3. [5][5] Lin Zhong, Qingshan Liu, Peng Yang, Bo Liu, Junzhou Huang, and Dimitris N Metaxas. Learning active facial patches for expression analysis. In: CVPR , 2012. -
  4. [6][6] C Fabian Benitez-Quiroz, Ramprakash Srinivasan, and Aleix M Martinez. Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: CVPR , 2016. -
  5. [7][7] Yong Li, Jiabei Zeng, Shiguang Shan, and Xilin Chen. Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE Trans. IP , 2018. -
  6. [14][14] Ronak Kosti, Jose M Alvarez, Adria Recasens, and Agata Lapedriza. Emotion recognition in context. In: CVPR , 2017. -
  7. [16][16] Jufeng Yang, Dongyu She, Yu-Kun Lai, Paul L Rosin, and Ming-Hsuan Yang. Weakly supervised coupled networks for visual sentiment analysis. In: CVPR , 2018. -
  8. [18][18] Stefanos Eleftheriadis, Ognjen Rudovic, and Maja Pantic. Discriminative shared gaussian processes for multiview and 19. view-invariant facial expression recognition. IEEE Trans. IP , 2015. -
  9. [20][20] Jiyoung Lee, Sunok Kim, Seungryong Kim, and Kwanghoon Sohn. Spatiotemporal attention based deep neural networks for emotion recognition. In: ICASSP , 2018. -
  10. [21][21] Mihalis A Nicolaou, Hatice Gunes, and Maja Pantic. Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Trans. AC , 2011. -
  11. [26][26] Ali Mollahosseini, Behzad Hasani, Michelle J Salvador, Hojjat Abdollahi, David Chan, and Mohammad H Mahoor. Facial expression recognition from world wild web. In: CVPR Work. , 2016. -
  12. [28][28] G. Patterson and J. Hays. Coco attributes: Attributes for people, animals, and objects. In: ECCV , 2016. -
  13. [30][30] A. Kleinsmith, N. Bianchi-Berthouze, and A. Steed. Automatic recognition of non-acted affective postures. IEEE Trans. Systems , 2011. -
  14. [32][32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. - · arXiv
  15. [34][34] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In: CVPR , 2016. -
  16. [35][35] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In: ICCV , 2017. -
  17. [36][36] Krishna Kumar Singh, Fanyi Xiao, and Yong Jae Lee. Track and transfer: Watching videos to simulate strong human supervision for weakly-supervised object detection. In: CVPR , 2016. -
  18. [38][38] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In: CVPR , 2018. -
  19. [39][39] Quanzeng You, Hailin Jin, and Jiebo Luo. Visual sentiment analysis by attending on local image regions. In: AAAI , 2017. -
  20. [42][42] Shikhar Sharma, Ryan Kiros, and Ruslan Salakhutdinov. Action recognition using visual attention. arXiv:1511.04119 , 2015. - · arXiv
  21. [43][43] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Trans. AC . -
  22. [44][44] Abhinav Dhall, Roland Goecke, Simon Lucey, Tom Gedeon, et al. Collecting large, richly annotated facial-expression databases from movies. IEEE Multi. , 2012. -
  23. [45][45] Yu-Gang Jiang, Baohan Xu, and Xiangyang Xue. Predicting emotions in user-generated videos. In: AAAI , 2014. -
  24. [48][48] Sunok Kim, Dongbo Min, Seungryong Kim, and Kwanghoon Sohn. Unified confidence estimation networks for robust stereo matching. IEEE Trans. IP , 2018. -
  25. [50][50] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In: CVPR , 2009. -

Cited by

Sorted by citations

Authors
Lee, J., Kim, S., Kim, S., Park, J., & Sohn, K.
Affiliations
Yonsei University (KR)
ORCID
Venue
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2019-10
Pages
10142-10151
DOI
10.1109/iccv.2019.01024
OpenAlex
W2968635608
Cited by
323
References
7
Citations by year
2026: 36 · 2025: 54 · 2024: 53 · 2023: 63 · 2022: 41 · 2021: 51 · 2020: 23 · 2019: 2
FWCI
36.88
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
context-aware emotion recognition, facial expression analysis, attention mechanism, feature fusion, two-stream network, multimodal emotion recognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review