Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?
Open accessAbstract
Despite being the appearance-based classifier of choice in recent years, relatively few works have examined how much convolutional neural networks (CNNs) can improve performance on accepted expression recognition benchmarks and, more importantly, examine what it is they actually learn. In this work, not only do we show that CNNs can achieve strong performance, but we also introduce an approach to decipher which portions of the face influence the CNN's predictions. First, we train a zero-bias CNN on facial expression data and achieve, to our knowledge, state-of-the-art performance on two expression recognition benchmarks: the extended Cohn-Kanade (CK+) dataset and the Toronto Face Dataset (TFD). We then qualitatively analyze the network by visualizing the spatial patterns that maximally excite different neurons in the convolutional layers and show how they resemble Facial Action Units (FAUs). Finally, we use the FAU labels provided in the CK+ dataset to verify that the FAUs observed in our filter visualizations indeed align with the subject's facial movements.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In benchmarking experiments across various datasets, many studies found that features extracted with CNNs lead to higher recognition accuracies than handcrafted features.
“In benchmarking experiments across various datasets, many studies found that features extracted with CNNs lead to higher recognition accuracies than handcrafted features (e.g., [87], [88], [89], [90], [91], [64], [92], [93], [94]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [87]
Studies running experiments on data augmentation report accuracy improvements (e.g., from 79% to 89% in [87]).
“Studies running experiments on this technique report accuracy improvements (e.g., from 79% to 89% in [87]; see also [74], [94]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [87]
Khorrami et al. [87] reported an increase in accuracy of 2.5% after applying dropout to fully-connected layers.
“Khorrami et al. [87] reported an increase in accuracy of 2.5% after applying dropout to fully-connected layers.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [87]
Multiple studies have found that higher layers of CNNs learn concepts similar to Action Units in facial expression recognition.
“Multiple studies have followed this approach to understand how CNNs recognize facial expressions, finding that higher layers learn concepts similar to Action Units (e.g., [87], [91]), validating previous research done in FER-Fig. 7 features some examples.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [87]
The citation lists haven’t been fetched yet
References 36 · Cited by 285
- Authors
- Khorrami, P., Paine, T. L., & Huang, T. S.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- ORCID
- Huang, Thomas S.
- Venue
- 2015 IEEE International Conference on Computer Vision Workshop (ICCVW)
- Publisher
- IEEE
- Published
- 2015-12
- Pages
- 19-27
- ISSN
- 2473-9936 (ISSN-L)
- OpenAlex
- W2195207531
- Cited by
- 285
- References
- 36
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 19 · 2023: 26 · 2022: 32 · 2021: 35 · 2020: 35 · 2019: 47 · 2018: 30 · 2017: 31 · 2016: 16
- FWCI
- 34.24
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Computer Vision Workshops (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Nvidia
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- facial expression recognition, facial action units, convolutional neural networks, interpretability, Cohn-Kanade dataset, facial motion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
