A performance comparison of eight commercially available automatic classifiers for facial affect recognition
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
In the wake of rapid advances in automatic affect analysis, commercial automatic classifiers for facial affect recognition have attracted considerable attention in recent years. While several options now exist to analyze dynamic video data, less is known about the relative performance of these classifiers, in particular when facial expressions are spontaneous rather than posed. In the present work, we tested eight out-of-the-box automatic classifiers, and compared their emotion recognition performance to that of human observers. A total of 937 videos were sampled from two large databases that conveyed the basic six emotions (happiness, sadness, anger, fear, surprise, and disgust) either in posed (BU-4DFE) or spontaneous (UT-Dallas) form. Results revealed a recognition advantage for human observers over automatic classification. Among the eight classifiers, there was considerable variance in recognition accuracy ranging from 48% to 62%. Subsequent analyses per type of expression revealed that performance by the two best performing classifiers approximated those of human observers, suggesting high agreement for posed expressions. However, classification accuracy was consistently lower (although above chance level) for spontaneous affective behavior. The findings indicate potential shortcomings of existing out-of-the-box classifiers for measuring emotions, and highlight the need for more spontaneous facial databases that can act as a benchmark in the training and testing of automatic emotion recognition systems. We further discuss some limitations of analyzing facial expressions that have been recorded in controlled environments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., et al. (2019) · Journal of Open Source Software · doi:10.21105/joss.01686 · Cited by 22,138 · cited 1×
“dplyr [80], forcats [81], ggplot2 [82], ggpubr [83], glue [84], gtable [85], here [86], irr [87], 461 kableExtra [88], knitr [89], lpSolve [90], magrittr [91], papaja [92], plotROC [93], pROC 462 [94], purrr [95], readr [96], stringr [97], tibble [98], tidyr [99], and tidyverse [100].”
Backgroundggplot2: Elegant Graphics for Data Analysis
Wickham, H. (2009) · doi:10.1007/978-0-387-98141-3 · Cited by 14,447
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697 · cited 1×
“dplyr [80], forcats [81], ggplot2 [82], ggpubr [83], glue [84], gtable [85], here [86], irr [87], 461 kableExtra [88], knitr [89], lpSolve [90], magrittr [91], papaja [92], plotROC [93], pROC 462 [94], purrr [95], readr [96], stringr [97], tibble [98], tidyr [99], and tidyverse [100].”
BackgroundAn argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 2×
“The emotions with which these expressions are associated are claimed to be basic, primary, or fundamental in the sense that they form the core emotional repertoire [6, 7].”
BackgroundA systematic analysis of performance measures for classification tasks
Sokolova, M., & Lapalme, G. (2009) · Information Processing & Management · doi:10.1016/j.ipm.2009.03.002 · Cited by 7,515
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559
The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding
EKMAN, P., & FRIESEN, W. V. (1969) · Semiotica · doi:10.1515/semi.1969.1.1.49 · Cited by 3,316
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 2×
“correspondence between the experience and expression of emotion [28].”
BackgroundEmotional expressions reconsidered: Challenges to inferring emotion from human facial movements
Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019) · Psychological Science in the Public Interest · doi:10.1177/1529100619832930 · Cited by 1,748
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · cited 1×
“by the face [19,20], post-hoc from video recordings as well as in real-time from a live 34 stream camera [21].”
MethodSeven-Year Outcomes in Ranibizumab-Treated Patients in ANCHOR, MARINA, and HORIZON
Rofagha, S., Bhisitkul, R. B., Boyer, D. S., Sadda, S. R., & Zhang, K. (2013) · Ophthalmology · doi:10.1016/j.ophtha.2013.03.046 · Cited by 1,023
Hess, U., & Blairy, S. (2001) · International Journal of Psychophysiology · doi:10.1016/s0167-8760(00)00161-6 · Cited by 697
The Structure of Emotion: Evidence From Neuroimaging Studies
Barrett, L. F., & Wager, T. D. (2006) · Current Directions in Psychological Science · doi:10.1111/j.0963-7214.2006.00411.x · Cited by 366
Meta-Analysis of the First Facial Expression Recognition Challenge
Valstar, M. F., Mehu, M., Jiang, B., Pantic, M., & Scherer, K. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2012.2200675 · Cited by 297 · cited 1×
“Expression Recognition and Analysis (FERA) challenge, emotion recognition by the top 61 performing algorithm was already being reported in 2011 at a rate of 84% [31].”
BackgroundLong-term outcomes of treatment of neovascular age-related macular degeneration
Gillies, M. C., Campain, A., Barthelmes, D., Simpson, J. M., Arnold, J. J., Guymer, R. H., McAllister, I. L., Essex, R. W., Morlet, N., & Hunyor, A. P. (2015) · Ophthalmology · doi:10.1016/j.ophtha.2015.05.010 · Cited by 254 · cited 4×
“Data from real world datasets also show that more than two-thirds of the patient population are no longer under review by an ophthalmologist 4 years after starting treatment.(5-7) There is broad agreement that visual acuity (VA) progressively worsens over time.”
BackgroundResultInfluentialMachine Analysis of Facial Expressions
Pantic, M., & Stewart, M. (2007) · Face Recognition · doi:10.5772/4847 · Cited by 248 · cited 2×
“Besides their limited ability to transfer to the subtlety and complexity of spontaneous recordings [43], the highly standardized form of prototypical expressions makes it difficult to generalize beyond specific training sets.”
BackgroundBuilding autonomous sensitive artificial listeners
Schroder, M., Bevacqua, E., Cowie, R., Eyben, F., Gunes, H., Heylen, D., ter Maat, M., McKeown, G., Pammi, S., Pantic, M., et al. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.34 · Cited by 185
Self-citationSpontaneous vs. posed facial behavior: automatic analysis of brow actions
Valstar, M. F., Pantic, M., Ambadar, Z., & Cohn, J. F. (2006) · Proceedings of the 8th international conference on Multimodal interfaces · doi:10.1145/1180995.1181031 · Cited by 168 · cited 1×
“Although 390 several efforts have been reported over the last few years on the automatic analysis of 391 spontaneous displays [55,70], most current automatic classifiers have typically been 392 trained and tested using posed or acted facial behavior.”
Background
Metadata
- Authors
- Dupré, D., Krumhuber, E. G., Küster, D., & McKeown, G. J.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-04-24
- Volume
- 15
- Issue
- 4
- Pages
- e0231968
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W3020447069
- Cited by
- 186
- References
- 20
- FWCI
- 25.68
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.