CoChat
Sign Up
PreprintCited by 210Emotion and Mood RecognitionFace recognition and analysisFace and Expression Recognition

Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace

Open access
Kollias, Dimitrios, Zafeiriou, Stefanos·2019·arXiv
DOI 10.48550/arxiv.1910.04855arXiv 1910.04855
Open this paper in CoChat — free accountOpen PDF

Abstract

Affective computing has been largely limited in terms of available data resources. The need to collect and annotate diverse in-the-wild datasets has become apparent with the rise of deep learning models, as the default approach to address any computer vision task. Some in-the-wild databases have been recently proposed. However: i) their size is small, ii) they are not audiovisual, iii) only a small part is manually annotated, iv) they contain a small number of subjects, or v) they are not annotated for all main behavior tasks (valence-arousal estimation, action unit detection and basic expression classification). To address these, we substantially extend the largest available in-the-wild database (Aff-Wild) to study continuous emotions such as valence and arousal. Furthermore, we annotate parts of the database with basic expressions and action units. As a consequence, for the first time, this allows the joint study of all three types of behavior states. We call this database Aff-Wild2. We conduct extensive experiments with CNN and CNN-RNN architectures that use visual and audio modalities; these networks are trained on Aff-Wild2 and their performance is then evaluated on 10 publicly available emotion databases. We show that the networks achieve state-of-the-art performance for the emotion recognition tasks. Additionally, we adapt the ArcFace loss function in the emotion recognition context and use it for training two new networks on Aff-Wild2 and then re-train them in a variety of diverse expression recognition databases. The networks are shown to improve the existing state-of-the-art. The database, emotion recognition models and source code are available at http://ibug.doc.ic.ac.uk/resources/aff-wild2.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1910.04855

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Aff-Wild2 contains 548 video clips collected from YouTube in real-world environments, with aligned faces and facial expression annotations, and eight classes of emotions.

    “we use the Aff-Wild2 dataset (Kollias and Zafeiriou, 2019; Kollias, 2022), which contains 548 video clips collected from YouTube in realworld environments. Each clip has several frames of aligned faces and each frame is annotated with a facial expression. It has eight classes of emotions (six basic emotions, neutral , and other ).”

    From A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations (2023) · cites this paper as Kollias and Zafeiriou, 2019

    The Aff-Wild2 database section (pp. 3–4) says the added videos came from YouTube and describes Aff-Wild2 as showing behaviour in real-world settings. But it reports 558 videos, not 548, and expression annotations for 84 videos in seven categories, not every frame of every video in eight categories. The cited passages do not establish that the faces are aligned. (LLM, conf=0.89)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 210

Research works printed but not linked: 48

  1. [1][1] Niki Aifanti, Christos Papachristou, and Anastasios Delopoulos. The mug facial expression database. In 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10 , pages 1-4. IEEE, 2010. -
  2. [2][2] C.F. Benitez-Quiroz, R. Srinivasan, and A.M. Martinez. Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In Proceedings of IEEE International Conference on Computer Vision & Pattern Recognition (CVPR'16) , Las Vegas, NV, USA, June 2016. -
  3. [3][3] Sanjay Bilakhia, Stavros Petridis, Anton Nijholt, and Maja Pantic. The mahnob mimicry database: A database of naturalistic human interactions. Pattern recognition letters , 66:52-61, 2015. -
  4. [4][4] Sumit Chopra, Raia Hadsell, Yann LeCun, et al. Learning a similarity metric discriminatively, with application to face verification. In CVPR (1) , pages 539546, 2005. -
  5. [5][5] Roddy Cowie and Randolph R Cornelius. Describing the emotional states that are expressed in speech. Speech communication , 40(1):5-32, 2003. -
  6. [6][6] Roddy Cowie, Ellen Douglas-Cowie, Susie Savvidou*, Edelle McMahon, Martin Sawey, and Marc Schröder. 'feeltrace': An instrument for recording perceived emotion in real time. In ISCA tutorial and research workshop (ITRW) on speech and emotion , 2000. -
  7. [7][7] Tim Dalgleish and Mick Power. Handbook of cognition and emotion . John Wiley &Sons, 2000. -
  8. [8][8] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. arXiv preprint arXiv:1801.07698 , 2018. - · arXiv
  9. [9][9] Wan Ding, Dong-Yan Huang, Zhuo Chen, Xinguo Yu, and Weisi Lin. Facial action recognition using very deep networks for highly imbalanced class distribution. In 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) , pages 1368-1372. IEEE, 2017. -
  10. [10][10] Shichuan Du, Yong Tao, and Aleix M Martinez. Compound facial expressions of emotion. Proceedings of the National Academy of Sciences , 111(15):E1454-E1462, 2014. -
  11. [12][12] Mariana-Iuliana Georgescu, Radu Tudor Ionescu, and Marius Popescu. Local learning with deep and handcrafted features for facial expression recognition. IEEE Access , 7:64827-64836, 2019. -
  12. [13][13] Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in representation learning: A report on three machine - learning contests. In International Conference on Neural Information Processing , pages 117-124. Springer, 2013. -
  13. [14][14] Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker. Multi-pie. Image and Vision Computing , 28(5):807-813, 2010. -
  14. [15][15] Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In International Workshop on Similarity-Based Pattern Recognition , pages 84-92. Springer, 2015. -
  15. [16][16] Dimitrios Kollias and Stefanos Zafeiriou. Aff-wild2: Extending the aff-wild database for affect recognition. arXiv preprint arXiv:1811.07770 , 2018. - · arXiv
  16. [17][17] Dimitrios Kollias and Stefanos Zafeiriou. A multi-task learning & generation framework: Valence-arousal, action units & primary expressions. arXiv preprint arXiv:1811.07771 , 2018. - · arXiv
  17. [18][18] Dimitrios Kollias and Stefanos Zafeiriou. Training deep neural networks with different datasets in-the-wild: The emotion recognition paradigm. In 2018 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2018. -
  18. [19][19] Dimitrios Kollias, Mihalis A Nicolaou, Irene Kotsia, Guoying Zhao, and Stefanos Zafeiriou. Recognition of affect in the wild using deep neural networks. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on , pages 1972-1979. IEEE, 2017. -
  19. [20][20] Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, Björn Schuller, Irene Kotsia, and Stefanos Zafeiriou. Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond. International Journal of Computer Vision , 127(6-7):907929, 2019. -
  20. [22][22] Shan Li, Weihong Deng, and JunPing Du. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2852-2861, 2017. -
  21. [23][23] Wei Li, Farnaz Abtahi, and Zhigang Zhu. Action unit detection with region adaptation, multi-labeling learning and optimal temporal fusing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 18411850, 2017. -
  22. [24][24] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 212-220, 2017. -
  23. [26][26] Michael J Lyons, Shigeru Akamatsu, Miyuki Kamachi, Jiro Gyoba, and Julien Budynek. The japanese female facial expression (jaffe) database. In Proceedings of third international conference on automatic face and gesture recognition , pages 1416, 1998. -
  24. [27][27] S Mohammad Mavadati, Mohammad H Mahoor, Kevin Bartlett, Philip Trinh, and Jeffrey F Cohn. Disfa: A spontaneous facial action intensity database. Affective Computing, IEEE Transactions on , 4(2):151-160, 2013. -
  25. [28][28] Gary McKeown, Michel Valstar, Roddy Cowie, Maja Pantic, and Marc Schroder. The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Transactions on Affective Computing , 3(1):5-17, 2011. -
  26. [29][29] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. arXiv preprint arXiv:1708.03985 , 2017. - · arXiv
  27. [30][30] Mahyar Najibi, Pouya Samangouei, Rama Chellappa, and Larry Davis. SSH: Single stage headless face detector. In The IEEE International Conference on Computer Vision (ICCV) , 2017. -
  28. [31][31] Maja Pantic, Michel Valstar, Ron Rademaker, and Ludo Maat. Web-based database for facial expression analysis. In Multimedia and Expo, 2005. ICME 2005. IEEE International Conference on , pages 5-pp. IEEE, 2005. -
  29. [32][32] Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In British Machine Vision Conference (BMVC) , 2015. -
  30. [33][33] Fabien Ringeval, Andreas Sonderegger, Jens Sauer, and Denis Lalanne. Introducing the recola multimodal corpus of remote collaborative and affective interactions. In Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on , pages 1-8. IEEE, 2013. -
  31. [34][34] Fabien Ringeval, Björn Schuller, Michel Valstar, Jonathan Gratch, Roddy Cowie, Stefan Scherer, Sharon Mozgai, Nicholas Cummins, Maximilian Schmi, and Maja Pantic. Avec 2017-real-life depression, and affect recognition workshop and challenge. 2017. -
  32. [35][35] James A Russell. Evidence of convergent validity on the dimensions of affect. Journal of personality and social psychology , 36(10):1152, 1978. -
  33. [36][36] Shankar Setty, Moula Husain, Parisa Beham, Jyothi Gudavalli, Menaka Kandasamy, Radhesyam Vaddi, Vidyagouri Hemadri, JC Karure, Raja Raju, B Rajan, et al. Indian movie face database: a benchmark for face recognition under wide variations. In 2013 Fourth National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics (NCVPRIPG) , pages 1-5. IEEE, 2013. -
  34. [37][37] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI , volume 4, page 12, 2017. -
  35. [38][38] Chuangao Tang, Wenming Zheng, Jingwei Yan, Qiang Li, Yang Li, Tong Zhang, and Zhen Cui. View-independent facial action unit detection. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) , pages 878-882. IEEE, 2017. -
  36. [40][40] Michel Valstar and Maja Pantic. Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In Proc. 3rd Intern. Workshop on EMOTION(satellite of LREC): Corpora for Research on Emotion and Affect , page 65, 2010. -
  37. [41][41] Michel F Valstar, Timur Almaev, Jeffrey M Girard, Gary McKeown, Marc Mehu, Lijun Yin, Maja Pantic, and Jeffrey F Cohn. Fera 2015-second facial expression recognition and analysis challenge. In Automatic Face and Gesture Recognition (FG), 2015 11th IEEE International Conference and Workshops on , volume 6, pages 1-8. IEEE, 2015. -
  38. [42][42] Michel F Valstar, Enrique Sánchez-Lozano, Jeffrey F Cohn, László A Jeni, Jeffrey M Girard, Zheng Zhang, Lijun Yin, and Maja Pantic. Fera 2017-addressing head pose in the third facial expression recognition and analysis challenge. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) , pages 839-847. IEEE, 2017. -
  39. [43][43] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5265-5274, 2018. -
  40. [44][44] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision , pages 499-515. Springer, 2016. -
  41. [45][45] CM Whissel. The dictionary of affect in language, emotion: Theory, research and experience: vol. 4, the measurement of emotions, r. Plutchik and H. Kellerman, Eds., New York: Academic , 1989. -
  42. [46][46] Shuo Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Wider face: A face detection benchmark. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016. -
  43. [47][47] Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang, and Matthew J Rosato. A 3d facial expression database for facial behavior research. In Automatic face and gesture recognition, 2006. FGR 2006. 7th international conference on , pages 211-216. IEEE, 2006. -
  44. [48][48] Lijun Yin, Xiaochen Chen, Yi Sun, Tony Worm, and Michael Reale. A highresolution 3d dynamic facial expression database. In Automatic Face & Gesture Recognition, 2008. FG'08. 8th IEEE International Conference On , pages 1-6. IEEE, 2008. -
  45. [49][49] Anıl Yüce, Hua Gao, and Jean-Philippe Thiran. Discriminant multi-label manifold embedding for facial action unit detection. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) , volume 6, pages 1-6. IEEE, 2015. -
  46. [50][50] Stefanos Zafeiriou, Dimitrios Kollias, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, and Irene Kotsia. Aff-wild: Valence and arousal'inthe-wild'challenge. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 34-41, 2017. -
  47. [51][51] Xing Zhang, Lijun Yin, Jeffrey F Cohn, Shaun Canavan, Michael Reale, Andy Horowitz, Peng Liu, and Jeffrey M Girard. Bp4d-spontaneous: a highresolution spontaneous 3d dynamic facial expression database. Image and Vision Computing , 32(10):692-706, 2014. -
  48. [52][52] Zheng Zhang, Jeff M Girard, Yue Wu, Xing Zhang, Peng Liu, Umur Ciftci, Shaun Canavan, Michael Reale, Andy Horowitz, Huiyuan Yang, et al. Multimodal spontaneous emotion corpus for human behavior analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3438-3446, 2016.

Authors
Kollias, D., & Zafeiriou, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Pages
297
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1910.04855
arXiv
1910.04855
OpenAlex
W4288102735
Cited by
210
References
0
Citations by year
2025: 8 · 2024: 28 · 2023: 47 · 2022: 33 · 2021: 57 · 2020: 32 · 2019: 3 · 2018: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Greenwich Academic Literature Archive (University of Greenwich) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Imperial College London
  • Engineering and Physical Sciences Research Council (EP/S010203/1)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
Aff-Wild2, affective computing, emotion recognition, valence-arousal estimation, facial expression recognition, action unit detection, in-the-wild datasets, audio-visual emotion recognition, multi-task learning, ArcFace loss, face recognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review