A Dynamic Texture-Based Approach to Recognition of Facial Actions and Their Temporal Models
Open accessAbstract
In this work, we propose a dynamic texture-based approach to the recognition of facial Action Units (AUs, atomic facial gestures) and their temporal models (i.e., sequences of temporal segments: neutral, onset, apex, and offset) in near-frontal-view face videos. Two approaches to modeling the dynamics and the appearance in the face region of an input video are compared: an extended version of Motion History Images and a novel method based on Nonrigid Registration using Free-Form Deformations (FFDs). The extracted motion representation is used to derive motion orientation histogram descriptors in both the spatial and temporal domain. Per AU, a combination of discriminative, frame-based GentleBoost ensemble learners and dynamic, generative Hidden Markov Models detects the presence of the AU in question and its temporal segments in an input image sequence. When tested for recognition of all 27 lower and upper face AUs, occurring alone or in combination in 264 sequences from the MMI facial expression database, the proposed method achieved an average event recognition accuracy of 89.2 percent for the MHI method and 94.3 percent for the FFD method. The generalization performance of the FFD method has been tested using the Cohn-Kanade database. Finally, we also explored the performance on spontaneous expressions in the Sensitive Artificial Listener data set.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Based on the observation that some AU are better detected using geometrical features and others using appearance ones, it was suggested that a combination of both might increase recognition performance.
“Based on the observation that some AU are better detected using geometrical features and others using appearance ones, it was suggested that a combination of both might increase recognition performance [127], [139], [183].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [127]
Increasing the number of recognized expressions usually follows two main directions: encoding expressions based on FACS action units instead of directly classifying them, and exploiting additional information provided by 3D facial data.
“Increasing the number of recognized expressions usually follows two main directions. First, expressions can be encoded based on FACS AUs [99], [113], [127], [139] instead of directly being classified. This provides a comprehensive coding of FEs without directly making a judgement on their intentionality. Other methods exploit additional information provided by 3D facial data.”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [127]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · see citing works
Recognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745 · see citing works
Cohen, I., Sebe, N., Gozman, F. G., Cirelo, M. C., & Huang, T. S. (2003) · 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. · doi:10.1109/cvpr.2003.1211408 · Cited by 155 · see citing works
A survey of affect recognition methods: audio, visual and spontaneous expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322216 · Cited by 144 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303 · see citing works
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
Static and dynamic 3D facial expression recognition: A comprehensive survey
Sandbach, G., Zafeiriou, S., Pantic, M., & Yin, L. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.005 · Cited by 357 · see citing works
Self-citationPan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142 · see citing works
The Machine Knows What You Are Hiding: An Automatic Micro-expression Recognition System
Wu, Q., Shen, X., & Fu, X. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24571-8_16 · Cited by 104 · see citing works
Partial least squares regression on grassmannian manifold for emotion recognition
Liu, M., Wang, R., Huang, Z., Shan, S., & Chen, X. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531738 · Cited by 62 · see citing works
Walecki, R., Rudovic, O., Pavlovic, V., & Pantic, M. (2015) · 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2015.7163137 · Cited by 65 · see citing works
Self-citationZhao, X., Dellandréa, E., Zou, J., & Chen, L. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.10.001 · Cited by 27 · see citing works
- Authors
- Koelstra, S., Pantic, M., & Patras, I.
- Affiliations
- Queen Mary University of London (GB)
- University of London (GB)
- ORCID
- Patras, Ioannis
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2010-11
- Volume
- 32
- Issue
- 11
- Pages
- 1940-1954
- ISSN
- 0162-8828 (ISSN-L)
- OpenAlex
- W2146780613
- Cited by
- 242
- References
- 4
- Citations by year
- 2025: 7 · 2024: 1 · 2023: 4 · 2022: 7 · 2021: 9 · 2020: 11 · 2019: 13 · 2018: 19 · 2017: 23 · 2016: 16 · 2015: 31 · 2014: 26 · 2013: 22 · 2012: 31
- FWCI
- 27.84
- Type
- journal article
- Language
- en
- Open access
- green
- research.utwente.nl
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- University of Twente Research Information (acceptedVersion)
- University of Twente Research Information (submittedVersion, other-oa)
- Funders
- University of Pittsburgh
- Engineering and Physical Sciences Research Council (EP/G033935/1, FP7/2007–2013)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Keywords
- facial action units, facial expression recognition, dynamic textures, motion history image, free-form deformation, non-rigid registration, hidden Markov models, spontaneous facial expressions
- MeSH terms
- Algorithms, Computing Methodologies, Face, Facial Expression, Gestures, Humans, Models, Biological, Pattern Recognition, Automated, Time Factors, Video Recording, Information Storage and Retrieval, Recognition, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
