EMOCA: Emotion Driven Monocular Face Capture and Animation
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
As 3D facial avatars become more widely used for communication, it is critical that they faithfully convey emotion. Unfortunately, the best recent methods that regress parametric 3D face models from monocular images are unable to capture the full spectrum of facial expression, such as subtle or extreme emotions. We find the standard reconstruction metrics used for training (landmark reprojection error, photometric error, and face recognition loss) are insufficient to capture high-fidelity expressions. The result is facial geometries that do not match the emotional content of the input image. We address this with EMOCA (EMOtion Capture and Animation), by introducing a novel deep perceptual emotion consistency loss during training, which helps ensure that the reconstructed 3D expression matches the expression depicted in the input image. While EMOCA achieves 3D reconstruction errors that are on par with the current best methods, it significantly outperforms them in terms of the quality of the reconstructed expression and the perceived emotional content. We also directly regress levels of valence and arousal and classify basic expressions from the estimated 3D face parameters. On the task of in-the-wild emotion recognition, our purely geometric approach is on par with the best image-based methods, highlighting the value of 3D geometry in analyzing human behavior. The model and code are publicly available at https://emoca.is.tue.mpg.de.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334 · cited 1×
“, happily surprised), continuous valence (positivenegative) and arousal (relaxed-intensive) values [71], or Facial Action Units (FACS) activations [25], where each action unit (AU) corresponds to a particular emotion-related facial muscle movement.”
BackgroundAn argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“Emotional states are commonly represented as discrete basic [23, 24] (e.”
BackgroundConstants across cultures in the face and emotion.
Ekman, P., & Friesen, W. V. (1971) · Journal of Personality and Social Psychology · doi:10.1037/h0030377 · Cited by 5,509 · cited 1×
“Emotional states are commonly represented as discrete basic [23, 24] (e.”
BackgroundLucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303 · cited 1×
“8, 14, 17, 21, 22 [56] Patrick Lucey, Jeffrey F.”
VGGFace2: A Dataset for Recognising Faces across Pose and Age
Cao, Q., Shen, L., Xie, W., Parkhi, O. M., & Zisserman, A. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00020 · Cited by 3,032
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Recognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745 · cited 1×
“Early work on expression recognition extracts geometric features defining shape and location of face components [59,85], appearance features [26,75], or combinations of these [39, Chapter 19].”
BackgroundRetinaFace: Single-Shot Multi-Level Face Localisation in the Wild
Deng, J., Guo, J., Ververas, E., Kotsia, I., & Zafeiriou, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00525 · Cited by 1,681
Compound facial expressions of emotion
Du, S., Tao, Y., & Martinez, A. M. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1322355111 · Cited by 806
DISFA: A Spontaneous Facial Action Intensity Database
Mavadati, S. M., Mahoor, M. H., Bartlett, K., Trinh, P., & Cohn, J. F. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2013.4 · Cited by 793
Towards Fast, Accurate and Stable 3D Dense Face Alignment
Guo, J., Zhu, X., Yang, Y., Yang, F., Lei, Z., & Li, S. Z. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58529-7_10 · Cited by 462
Expert system for automatic analysis of facial expressions
Pantic, M., & Rothkrantz, L. J. M. (2000) · Image and Vision Computing · doi:10.1016/s0262-8856(00)00034-2 · Cited by 415
Static and dynamic 3D facial expression recognition: A comprehensive survey
Sandbach, G., Zafeiriou, S., Pantic, M., & Yin, L. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.005 · Cited by 357 · cited 1×
“While early work like Wen and Huang [94] uses 3D non-rigid surface tracking to extract features for expression reconstruction, the majority of 3D-based methods focus on recognizing expressions from 3D scans [58, 72].”
BackgroundRacial faces in the wild: Reducing racial bias by information maximization adaptation network
Wang, M., Deng, W., Hu, J., Tao, X., & Huang, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00078 · Cited by 353 · cited 1×
“Large image datasets used by previous 3D reconstruction methods, while containing a large number of subjects of diverse ethnicities, lack emotional expressivity [14, 17, 44, 92].”
MethodSewa db: A rich database for audio-visual emotion and sentiment research in the wild
Kossaifi, J., Walecki, R., Panagakis, Y., Shen, J., Schmitt, M., Ringeval, F., Han, J., Pandit, V., Toisoul, A., Schuller, B., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2944808 · Cited by 243
AFEW-VA database for valence and arousal estimation in-the-wild
Kossaifi, J., Tzimiropoulos, G., Todorovic, S., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.02.001 · Cited by 236 · cited 1×
“We evaluate emotion recognition on the AffectNet test set [57] and the AFEW-VA test set [47].”
MethodHuman Facial Expression Recognition using a 3D Morphable Model
Ramanathan, S., Kassim, A., Venkatesh, Y. V., & Wah, W. S. (2006) · 2006 International Conference on Image Processing · doi:10.1109/icip.2006.312417 · Cited by 62 · cited 1×
“Among these, the most relevant to EMOCA, is [65] as they use 3DMM features to classify three expressions (obtained by fitting the 3DMM to the scans); most other methods use diverse 2D and 3D features extracted from the textured 3D scans.”
Method
Metadata
- Authors
- Danecek, R., Black, M., & Bolkart, T.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 20279-20290
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4312590328
- Cited by
- 238
- References
- 22
- FWCI
- 17.87
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.