The DOI matches its Crossref record; 84 of the 84 references it cites were found in a registry or an index.
Fully Automatic Recognition of the Temporal Phases of Facial Actions
Abstract
Past work on automatic analysis of facial expressions has focused mostly on detecting prototypic expressions of basic emotions like happiness and anger. The method proposed here enables the detection of a much larger range of facial behavior by recognizing facial muscle actions [action units (AUs)] that compound expressions. AUs are agnostic, leaving the inference about conveyed intent to higher order decision making (e.g., emotion recognition). The proposed fully automatic method not only allows the recognition of 22 AUs but also explicitly models their temporal characteristics (i.e., sequences of temporal segments: neutral, onset, apex, and offset). To do so, it uses a facial point detector based on Gabor-feature-based boosted classifiers to automatically localize 20 facial fiducial points. These points are tracked through a sequence of images using a method called particle filtering with factorized likelihoods. To encode AUs and their temporal activation models based on the tracking data, it applies a combination of GentleBoost, support vector machines, and hidden Markov models. We attain an average AU recognition rate of 95.3% when tested on a benchmark set of deliberately displayed facial expressions and 72% when tested on spontaneous expressions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
A New Approach to Linear Filtering and Prediction Problems
Kalman, R. E. (1960) · Journal of Basic Engineering · doi:10.1115/1.3662552 · Cited by 31,751
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
An Introduction to Support Vector Machines and Other Kernel-based Learning Methods
Cristianini, N., & Shawe-Taylor, J. (2000) · doi:10.1017/cbo9780511801389 · Cited by 13,855
(1981) · Cited by 11,963
Friedman, J., Hastie, T., & Tibshirani, R. (2000) · The Annals of Statistics · doi:10.1214/aos/1016218223 · Cited by 6,884
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,388
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
Performance of optical flow techniques
Barron, J. L., Fleet, D. J., & Beauchemin, S. S. (1994) · International Journal of Computer Vision · doi:10.1007/bf01420984 · Cited by 4,305
Detecting faces in images: a survey
Yang, M.-H., Kriegman, D. J., & Ahuja, N. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.982883 · Cited by 3,427
The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding
EKMAN, P., & FRIESEN, W. V. (1969) · Semiotica · doi:10.1515/semi.1969.1.1.49 · Cited by 3,316
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Comprehensive database for facial expression analysis
Kanade, T., Cohn, J. F., & Tian, Y. (2002) · Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) · doi:10.1109/afgr.2000.840611 · Cited by 2,591
Filtering via Simulation: Auxiliary Particle Filters
Pitt, M. K., & Shephard, N. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10474153 · Cited by 2,438
A Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205
Ekman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,941
(2011) · Springer eBooks · doi:10.1007/978-0-85729-932-1 · Cited by 1,819
Recognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745
Platt, J. C. (2000) · Advances in Large-Margin Classifiers · doi:10.7551/mitpress/1113.003.0008 · Cited by 1,156
Web-Based Database for Facial Expression Analysis
Pantic, M., Valstar, M., Rademaker, R., & Maat, L. (2005) · 2005 IEEE International Conference on Multimedia and Expo · doi:10.1109/icme.2005.1521424 · Cited by 1,039
Self-citationDonato, G., Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.799905 · Cited by 999
Hjelmås, E., & Low, B. K. (2001) · Computer Vision and Image Understanding · doi:10.1006/cviu.2001.0921 · Cited by 919
Toward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791
Self-citation(2005) · Springer eBooks · doi:10.1007/b138828 · Cited by 789
Kotsia, I., & Pitas, I. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.884954 · Cited by 687
Ambadar, Z., Schooler, J. W., & Cohn, J. F. (2005) · Psychological Science · doi:10.1111/j.0956-7976.2005.01548.x · Cited by 656
The Psychology of Facial Expression
Mandler, G. (1997) · Cambridge University Press eBooks · doi:10.1017/cbo9780511659911 · Cited by 655
Darwin, Deception, and Facial Expression
EKMAN, P. (2003) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1280.010 · Cited by 624
Facial expression of pain: An evolutionary account
Williams, A. C. D. C. (2002) · Behavioral and Brain Sciences · doi:10.1017/s0140525x02000080 · Cited by 583
Pantic, M., & Patras, I. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.859075 · Cited by 580
Self-citationActive and dynamic information fusion for facial expression understanding from image sequences
Zhang, Y., & Ji, Q. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.93 · Cited by 461
Measuring facial expressions by computer image analysis
Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999) · Psychophysiology · doi:10.1017/s0048577299971664 · Cited by 424
Facial Action Recognition for Facial Expression Analysis From Static Face Images
Pantic, M., & Rothkrantz, L. J. M. (2004) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2004.825931 · Cited by 417
Self-citationReal-time combined 2D+3D active appearance models
Xiao, J., Baker, S., Matthews, I., & Kanade, T. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315210 · Cited by 416
Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships
Tong, Y., Liao, W., & Ji, Q. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1094 · Cited by 378
Dynamics of facial expression extracted automatically from video
Littlewort, G., Bartlett, M. S., Fasel, I., Susskind, J., & Movellan, J. (2006) · Image and Vision Computing · doi:10.1016/j.imavis.2005.09.011 · Cited by 358
The painful face – Pain expression recognition using active appearance models
Ashraf, A. B., Lucey, S., Cohn, J. F., Chen, T., Ambadar, Z., Prkachin, K. M., & Solomon, P. E. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2009.05.007 · Cited by 355
A real-time automated system for the recognition of human facial expressions
Anderson, K., & McOwan, P. W. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.854502 · Cited by 355
Fully Automatic Facial Action Unit Detection and Temporal Analysis
Valstar, M., & Pantic, M. (2006) · 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06) · doi:10.1109/cvprw.2006.85 · Cited by 346
Self-citationParticle Methods for Change Detection, System Identification, and Control
Andrieu, C., Doucet, A., Singh, S. S., & Tadic, V. B. (2004) · Proceedings of the IEEE · doi:10.1109/jproc.2003.823142 · Cited by 335
THE TIMING OF FACIAL MOTION IN POSED AND SPONTANEOUS SMILES
COHN, J. F., & SCHMIDT, K. L. (2004) · International Journal of Wavelets, Multiresolution and Information Processing · doi:10.1142/s021969130400041x · Cited by 298
Fully Automatic Facial Action Recognition in Spontaneous Behavior
Bartlett, M. S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., & Movellan, J. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.55 · Cited by 290
Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures
El Kaliouby, R., & Robinson, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.427 · Cited by 282
Fully Automatic Facial Feature Point Detection Using Gabor Feature Based Boosted Classifiers
Vukadinovic, D., & Pantic, M. (2006) · 2005 IEEE International Conference on Systems, Man and Cybernetics · doi:10.1109/icsmc.2005.1571392 · Cited by 263
Self-citationMcCane, B., Novins, K., Crannitch, D., & Galvin, B. (2001) · Computer Vision and Image Understanding · doi:10.1006/cviu.2001.0930 · Cited by 250
Machine Analysis of Facial Expressions
Pantic, M., & Stewart, M. (2007) · Face Recognition · doi:10.5772/4847 · Cited by 248
Action unit detection using sparse appearance descriptors in space-time video volumes
Jiang, B., Valstar, M. F., & Pantic, M. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771416 · Cited by 233
Self-citationA generative framework for real time object detection and classification
Fasel, I., Fortenberry, B., & Movellan, J. (2005) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2004.07.014 · Cited by 208
How to distinguish posed from spontaneous smiles using geometric features
Valstar, M. F., Gunes, H., & Pantic, M. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322202 · Cited by 203
Self-citationFaces of pain: automated measurement of spontaneousallfacial expressions of genuine and posed pain
Littlewort, G. C., Bartlett, M. S., & Lee, K. (2007) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1322192.1322198 · Cited by 172
Spontaneous vs. posed facial behavior: automatic analysis of brow actions
Valstar, M. F., Pantic, M., Ambadar, Z., & Cohn, J. F. (2006) · Proceedings of the 8th international conference on Multimodal interfaces · doi:10.1145/1180995.1181031 · Cited by 168
Self-citationHaar Features for FACS AU Recognition
Whitehill, J., & Omlin, C. W. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.61 · Cited by 162
Valstar, M., & Pantić, M. (2007) · Springer eBooks · doi:10.1007/978-3-540-75773-3_13 · Cited by 160
Self-citationValstar, M., Patras, I., & Pantić, M. (2006) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvpr.2005.457 · Cited by 160
Self-citationYoung, A. W. (1998) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780198524205.001.0001 · Cited by 135
The painful face: pain expression recognition using active appearance models
Ashraf, A. B., Lucey, S., Cohn, J. F., Chen, T., Ambadar, Z., Prkachin, K., Solomon, P., & Theobald, B. J. (2007) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1322192.1322197 · Cited by 128
Cohn, J. F., & Ekman, P. (2005) · Oxford University Press eBooks · doi:10.1093/oso/9780198529613.003.0002 · Cited by 111
Foundations of human computing: facial expression and emotion
Cohn, J. F. (2006) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1180995.1181043 · Cited by 107
Hierarchical wavelet networks for facial feature localization
Feris, R. S., Gemmell, J., Toyama, K., & Krüger, V. (2003) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2002.1004143 · Cited by 106
Boosting encoded dynamic features for facial expression recognition
Yang, P., Liu, Q., & Metaxas, D. (2008) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.03.014 · Cited by 100
Particle filtering with factorized likelihoods for tracking facial features
Patras, I., & Pantić, M. (2004) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2004.1301515 · Cited by 100
Bartlett, M. S., Viola, P. A., B, T. J. S. A., Golomb, B. A., Larsen, J. O., Hager, J. C., & Ekman, P. (1995) · Cited by 97
Subtly different facial expression recognition and expression intensity estimation
Lien, J. J.-J., Takeo Kanade, A., Cohn, J. F., & Li, C.-C. (2002) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1998.698704 · Cited by 97
Detecting Facial Actions and their Temporal Segments in Nearly Frontal-View Face Image Sequences
Pantić, M., & Patras, I. (2006) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2005.1571665 · Cited by 93
Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior
Cohn, J. F., Reed, L. I., Ambadar, Z., Xiao, J., & Moriyama, T. (2005) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2004.1398367 · Cited by 91
Effects of Alcohol on Group Formation Among Male Social Drinkers
Kirchner, T., Sayette, M. A., Cohn, J. F., et al. (2006) · Journal of Studies on Alcohol · doi:10.15288/jsa.2006.67.785 · Cited by 82
Foundations of Human Computing: Facial Expression and Emotion
Cohn, J. F. (2007) · Springer eBooks · doi:10.1007/978-3-540-72348-6_1 · Cited by 77
A comparison of shape constrained facial feature detectors
Cristinacce, D., & Cootes, T. F. (2004) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2004.1301561 · Cited by 73
7th International Conference on Automatic Face and Gesture Recognition (FGR06)
Gralewski, L., Campbell, N., & Penton‐Voak, I. S. (2006) · The Institute of Electrical and Electronics Engineers · Cited by 69
Koelstra, S., & Pantić, M. (2008) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2008.4813361 · Cited by 60
Biologically vs. Logic Inspired Encoding of Facial Actions and Emotions in Video
Valstar, M., & Pantić, M. (2006) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2006.262464 · Cited by 56
Self-citationRosenberg, E. L., Ekman, P., & Blumenthal, J. A. (1998) · Health Psychology · doi:10.1037//0278-6133.17.4.376 · Cited by 48
Speech recognition with support vector machines in a hybrid system
Krüger, S. E., Schafföner, M., Katz, M., Andelic, E., & Wendemuth, A. (2005) · Interspeech (USB) · doi:10.21437/interspeech.2005-237 · Cited by 45
The components of conversational facial expressions
Cunningham, D. W., Kleiner, M., Bülthoff, H., & Wallraven, C. (2004) · Symposium on Applied Perception in Graphics and Visualization (APGV) · doi:10.1145/1012551.1012578 · Cited by 44
Information extraction from image sequences of real-world facial expressions
Gu, H., & Ji, Q. (2004) · Machine Vision and Applications · doi:10.1007/s00138-004-0161-6 · Cited by 30
3D shape constraint for facial feature localization using probabilistic-like output
Chen, L., Zhang, L., Zhang, H., & Abdel-Mottaleb, M. S. (2004) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2004.1301548 · Cited by 30
Cohen, M. M. (2006) · doi:10.1093/oso/9780195300406.001.0001 · Cited by 28
Automatic Facial Point Detection
Holden, E., & Owens, R. A. (2002) · UWA Profiles and Research Repository (University of Western Australia) · Cited by 27
Using a Tensor Framework for the Analysis of Facial Dynamics
Gralewski, L., Campbell, N., & Penton‐Voak, I. S. (2006) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fgr.2006.108 · Cited by 23
Pantić, M. (2006) · Lecture notes in computer science · doi:10.1007/11825890_2 · Cited by 22
Facial action tracking using particle filters and active appearance models
Hamlaoui, S., & Davoine, F. (2005) · doi:10.1145/1107548.1107592 · Cited by 15
Patras, I., & Pantić, M. (2006) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2005.1571287 · Cited by 14
AUTOMATION OF NON-VERBAL COMMUNICATION OF FACIAL EXPRESSIONS
Pantić, M., Rothkrantz, L. J. M., & Koppelaar, H. (2007) · Cited by 14
Facial Action Coding Using Multiple Visual Cues and a Hierarchy of Particle Filters
McCall, J. C., & Trivedi, M. M. (2006) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2006.77 · Cited by 9
Cited by
Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition
Sariyanidi, E., Gunes, H., & Cavallaro, A. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2366127 · Cited by 685
Static and dynamic 3D facial expression recognition: A comprehensive survey
Sandbach, G., Zafeiriou, S., Pantic, M., & Yin, L. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.005 · Cited by 357
Automatic Analysis of Facial Actions: A Survey
Martinez, B., Valstar, M. F., Jiang, B., & Pantic, M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2731763 · Cited by 350
Self-citationTelemental Health Care, an Effective Alternative to Conventional Mental Care: a Systematic Review
Langarizadeh, M., Tabatabaei, M., Tavakol, K., Naghipour, M., & Moghbeli, F. (2017) · Acta Informatica Medica · doi:10.5455/aim.2017.25.240-246 · Cited by 314
Selective Transfer Machine for Personalized Facial Action Unit Detection
Chu, W.-S., De la Torre, F., & Cohn, J. F. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.451 · Cited by 307
Metadata
- Authors
- Valstar, M. F., & Pantic, M.
- Venue
- IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)
- Published
- 2012-02
- Volume
- 42
- Issue
- 1
- Pages
- 28-43
- ISSN
- 1083-4419 (ISSN-L)
- 1941-0492
- OpenAlex
- W2025216571
- Cited by
- 304
- References
- 84
- FWCI
- 28.64
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.