The DOI matches its Crossref record; 228 of the 228 references it cites were found in a registry or an index.
Automatic Analysis of Facial Actions: A Survey
Open accessAbstract
As one of the most comprehensive and objective ways to describe facial expressions, the Facial Action Coding System (FACS) has recently received significant attention. Over the past 30 years, extensive research has been conducted by psychologists and neuroscientists on various aspects of facial expression analysis using FACS. Automating FACS coding would make this research faster and more widely applicable, opening up new avenues to understanding how we communicate through facial expressions. Such an automated process can also potentially increase the reliability, precision and temporal resolution of coding. This paper provides a comprehensive survey of research into machine analysis of facial actions. We systematically review all components of such systems: pre-processing, feature extraction and machine coding of facial actions. In addition, the existing FACS-coded facial expression databases are summarised. Finally, challenges that have to be addressed to make automatic facial action analysis applicable in real-life situations are extensively discussed. There are two underlying motivations for us to write this survey paper: the first is to provide an up-to-date review of the existing literature, and the second is to offer some insights into the future of machine recognition of facial actions: what are the challenges and opportunities that researchers in the field face.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–228 of 228
Kernel Conditional Ordinal Random Fields for Temporal Segmentation of Facial Action Units
Rudovic, O. O., Pavlović, V., & Pantić, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33868-7_26 · Cited by 35
RAPS: Robust and Efficient Automatic Construction of Person-Specific Deformable Models
Sagonas, C., Panagakis, Y., Zafeiriou, S. P., & Pantić, M. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.231 · Cited by 35
Automatic detection of non-posed facial action units
Mavadati, S. M., Mahoor, M. H., Bartlett, K., et al. (2012) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2012.6467235 · Cited by 35
View-Independent Facial Action Unit Detection
Tang, C., Zheng, W., Yan, J., Li, Q., Li, Y., Zhang, T., & Cui, Z. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.113 · Cited by 34
Ashraf, A. B., Lucey, S., & Chen, T. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.75 · Cited by 33
Spatio-temporal Event Classification Using Time-Series Kernel Based Structured Sparsity
Jeni, L. A., Lörincz, A., Szabó, Z., Cohn, J. F., & Takeo Kanade, A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10593-2_10 · Cited by 30
Empirical analysis of cascade deformable models for multi-view face detection
Orozco, J., Martínez, B., & Pantić, M. (2015) · Image and Vision Computing · doi:10.1016/j.imavis.2015.07.002 · Cited by 29
Multiple-Facial Action Unit Recognition by Shared Feature Learning and Semantic Relation Modeling
Zhu, Y., Wang, S., Yue, L., & Ji, Q. (2014) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2014.293 · Cited by 26
Intensity Estimation of Spontaneous Facial Action Units Based on Their Sparsity Properties
Mohammadi, M. R., Fatemizadeh, E., & Mahoor, M. H. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2416317 · Cited by 25
Dynamic facial expression recognition by joint static and multi-time gap transition classification
Dapogny, A., Bailly, K., & Dubuisson, S. (2015) · 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2015.7163111 · Cited by 24
The Detection of Concept Frames Using Clustering Multi-instance Learning
Tax, D. M. J., Hendriks, E. A., Valstar, M., & Pantić, M. (2010) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2010.715 · Cited by 23
Self-citationContext-Independent Facial Action Unit Recognition Using Shape and Gabor Phase Information
González, I., Hichem Sahli, B., Enescu, V., & Verhelst, W. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24600-5_58 · Cited by 22
Regularized Multi-Concept MIL for weakly-supervised facial behavior categorization
Ruiz, A., Binefa, X., & van de Weijer, J. (2014) · British Machine Vision Conference (BMVC) · doi:10.5244/c.28.13 · Cited by 21
Facial action unit detection using kernel partial least squares
Gehrig, T., & Ekenel, H. K. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130506 · Cited by 21
Simultaneous Detection of Multiple Facial Action Units via Hierarchical Task Structure Learning
Zhang, X. L., & Mahoor, M. H. (2014) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2014.326 · Cited by 21
A practical approach to real-time neutral feature subtraction for facial expression recognition
Haber, N., Voss, C., Fazel, A., Winograd, T. A., & Wall, D. P. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477675 · Cited by 21
Learning to combine local models for facial Action Unit detection
Jaiswal, S., Martínez, B., & Valstar, M. (2015) · IEEE International Conference on Automatic Face & Gesture Recognition · doi:10.1109/fg.2015.7284872 · Cited by 20
Self-citationDynamic soft encoded patterns for facial event analysis
Yang, P., Liu, Q., & Metaxas, D. (2011) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2010.11.015 · Cited by 16
Temporal modeling of facial actions from face profile image sequences
Pantić, M., & Patras, I. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2004.1394122 · Cited by 16
Unsupervised Synchrony Discovery in Human Interaction
Chu, W.-S., Zeng, J., la Torre, F. D., Cohn, J. F., & Messinger, D. S. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.360 · Cited by 16
Lin, J., Wu, C., & Wei, W.-L. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638305 · Cited by 15
Face reading from speech — predicting facial action units from audio cues
Ringeval, F., Marchi, E., Méhu, M., Scherer, K. R., & Schuller, B. W. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-435 · Cited by 15
AUTOMATION OF NON-VERBAL COMMUNICATION OF FACIAL EXPRESSIONS
Pantić, M., Rothkrantz, L. J. M., & Koppelaar, H. (2007) · Cited by 14
Temporal Facial Expression Modeling for Automated Action Unit Intensity Measurement
Mavadati, S. M., & Mahoor, M. H. (2014) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2014.795 · Cited by 13
Facial Action Units Detection with Multi-Features and -AUs Fusion
Li, X., Chen, S., & Jin, Q. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.110 · Cited by 13
Amirian, M., Kächele, M., Palm, G., & Schwenker, F. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.109 · Cited by 13
Facial Actions as Social Signals
Valstar, M., Zafeiriou, S. P., & Pantić, M. (2017) · Cambridge University Press eBooks · doi:10.1017/9781316676202.011 · Cited by 7
Self-citationFacial Action Recognition in 2D and 3D
Valstar, M., Zafeiriou, S. P., & Pantić, M. (2014) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-4666-5966-7.ch008 · Cited by 3
Self-citation
Cited by
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 1,753
Facial Expression Analysis under Partial Occlusion: A Survey
Zhang, L., Verma, B., Tjondronegoro, D., & Chandran, V. (2019) · ACM Computing Surveys · doi:10.1145/3158369 · Cited by 1,258
Engagement detection in online learning: a review
Dewan, M. A. A., Murshed, M., & Lin, F. (2019) · Smart Learning Environments · doi:10.1186/s40561-018-0080-z · Cited by 350
Island Loss for Learning Discriminative Features in Facial Expression Recognition
Cai, J., Meng, Z., Khan, A. S., Li, Z., OReilly, J., & Tong, Y. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00051 · Cited by 321
Kim, J.-H., Kim, B.-G., Roy, P. P., & Jeong, D.-M. (2019) · IEEE Access · doi:10.1109/access.2019.2907327 · Cited by 303
Metadata
- Authors
- Martinez, B., Valstar, M. F., Jiang, B., & Pantic, M.
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2019-07-01
- Volume
- 10
- Issue
- 3
- Pages
- 325-347
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W2737559518
- Cited by
- 350
- References
- 228
- FWCI
- 42.80
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.