Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences
Open accessAbstract
Automatic analysis of human facial expression is a challenging problem with many applications. Most of the existing automated systems for facial expression analysis attempt to recognize a few prototypic emotional expressions, such as anger and happiness. Instead of representing another approach to machine analysis of prototypic facial expressions of emotion, the method presented in this paper attempts to handle a large range of human facial behavior by recognizing facial muscle actions that produce expressions. Virtually all of the existing vision systems for facial muscle action detection deal only with frontal-view face images and cannot handle temporal dynamics of facial actions. In this paper, we present a system for automatic recognition of facial action units (AUs) and their temporal models from long, profile-view face image sequences. We exploit particle filtering to track 15 facial points in an input face-profile sequence, and we introduce facial-action-dynamics recognition from continuous video input using temporal rules. The algorithm performs both automatic segmentation of an input video into facial expressions pictured and recognition of temporal segments (i.e., onset, apex, offset) of 27 AUs occurring alone or in a combination in the input face-profile video. A recognition rate of 87% is achieved.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MMI introduced temporal labeling of onset, apex and offset.
“It also introduced temporal labeling of onset, apex and offset.”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [114]
The MMI dataset adds profile views of not only the primary expressions but most of the AU of the FACS system.
“It adds profile views of not only the primary expressions but most of the AU of the FACS system.”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [114]
In later years various ways of dynamic representation were explored.
“In later years various ways of dynamic representation were also explored like tracking geometrical deformations across frames in RGB [114], [116] and 3D [119], [128] or directly extracting features from RGB [127] and thermal frame sequences [196], [210].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [114]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Huber, P. J. (1981) · Wiley Series in Probability and Statistics · doi:10.1002/0471725250 · Cited by 7,066 · see citing works
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · cited 1× · see citing works
“Few methods have been reported for automatic AU detection in face image sequences [16].”
MethodRecognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745 · see citing works
Measuring facial expressions by computer image analysis
Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999) · Psychophysiology · doi:10.1017/s0048577299971664 · Cited by 424 · see citing works
Affective multimodal human-computer interaction
Pantic, M., Sebe, N., Cohn, J. F., & Huang, T. (2005) · Proceedings of the 13th annual ACM international conference on Multimedia · doi:10.1145/1101149.1101299 · Cited by 234 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300 · see citing works
Social signal processing: Survey of an emerging domain
Vinciarelli, A., Pantic, M., & Bourlard, H. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.11.007 · Cited by 842 · see citing works
Self-citationCorneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
The painful face – Pain expression recognition using active appearance models
Ashraf, A. B., Lucey, S., Cohn, J. F., Chen, T., Ambadar, Z., Prkachin, K. M., & Solomon, P. E. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2009.05.007 · Cited by 355 · see citing works
Machine Analysis of Facial Expressions
Pantic, M., & Stewart, M. (2007) · Face Recognition · doi:10.5772/4847 · Cited by 248 · see citing works
Self-citationMeng, H., Huang, D., Wang, H., Yang, H., AI-Shuraifi, M., & Wang, Y. (2013) · Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge · doi:10.1145/2512530.2512532 · Cited by 225 · see citing works
Spontaneous vs. posed facial behavior: automatic analysis of brow actions
Valstar, M. F., Pantic, M., Ambadar, Z., & Cohn, J. F. (2006) · Proceedings of the 8th international conference on Multimodal interfaces · doi:10.1145/1180995.1181031 · Cited by 168 · see citing works
A survey of affect recognition methods: audio, visual and spontaneous expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322216 · Cited by 144 · see citing works
Real-time 2D+3D facial action and expression recognition
Tsalakanidou, F., & Malassiotis, S. (2010) · Pattern Recognition · doi:10.1016/j.patcog.2009.12.009 · Cited by 125 · see citing works
Facial expression recognition via deep learning
Lv, Y., Feng, Z., & Xu, C. (2014) · 2014 International Conference on Smart Computing · doi:10.1109/smartcomp.2014.7043872 · Cited by 122 · see citing works
Chen, S., Tian, Y., Liu, Q., & Metaxas, D. N. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.014 · Cited by 72 · see citing works
Expression Recognition Method Based on a Lightweight Convolutional Neural Network
Zhao, G., Yang, H., & Yu, M. (2020) · IEEE Access · doi:10.1109/access.2020.2964752 · Cited by 53 · see citing works
Zhao, X., Dellandréa, E., Zou, J., & Chen, L. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.10.001 · Cited by 27 · see citing works
- Authors
- Pantic, M., & Patras, I.
- Affiliations
- University of York (GB)
- ORCID
- Patras, I.
- Venue
- IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics)
- Published
- 2006-04
- Volume
- 36
- Issue
- 2
- Pages
- 433-449
- ISSN
- 1083-4419 (ISSN-L)
- OpenAlex
- W2110885456
- Cited by
- 580
- References
- 5
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 10 · 2023: 7 · 2022: 13 · 2021: 33 · 2020: 24 · 2019: 54 · 2018: 31 · 2017: 32 · 2016: 32 · 2015: 48 · 2014: 44 · 2013: 41 · 2012: 42
- FWCI
- 35.40
- Type
- journal article
- Language
- en
- Open access
- green
- resolver.tudelft.nl
- Licence
- other-oa
- Locations
- IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) (publishedVersion)
- PubMed (publishedVersion)
- Research Repository (Delft University of Technology) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- facial expression analysis, facial action units, particle filter, temporal rules
- MeSH terms
- Algorithms, Artificial Intelligence, Face, Facial Expression, Humans, Image Enhancement, Image Interpretation, Computer-Assisted, Movement, Pattern Recognition, Automated, Photography, Sensitivity and Specificity, Subtraction Technique, Time Factors, Video Recording, Reproducibility of Results, Cluster Analysis, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
