Dynamics of Facial Expression Extracted Automatically from Video
Abstract
We present a systematic comparison of machine learning methods applied to the problem of fully automatic recognition of facial expressions, including AdaBoost, support vector machines, and linear discriminant analysis. Each video-frame is first scanned in real-time to detect approximately upright-frontal faces. The faces found are scaled into image patches of equal size, convolved with a bank of Gabor energy filters, and then passed to a recognition engine that codes facial expressions into 7 dimensions in real time: neutral, anger, disgust, fear, joy, sadness, surprise. We report results on a series of experiments comparing spatial frequency ranges, feature selection techniques, and recognition engines. Best results were obtained by selecting a subset of Gabor filters using AdaBoost and then training Support Vector Machines on the outputs of the filters selected by AdaBoost. The generalization performance to new subjects for a 7-way forced choice was 93% or more correct on two publicly available datasets, the best performance reported so far on these datasets. Surprisingly, registration of internal facial features was not necessary, even though the face detector does not provide precisely registered images. The outputs of the classifier change smoothly as a function of time and thus can be used for unobtrusive motion capture. We developed an end-to-end system that provides facial expression codes at 24 frames per second and animates a computer generated character. In real-time this expression mirror operates down to resolutions of 16 pixels from eye to eye. We also applied the system to fully automated facial action coding.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the first part of the decade static representations were the primary choice in RGB, 3D and thermal.
“In the first part of the decade static representations were the primary choice in both RGB [99], [139], 3D [118], [120], [125], [131], [133], [219] and thermal [111].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [99]
Others focused on appearance representations like Gabor filters, optical flow and LBPs or combinations.
“Others focused on appearance representations like Gabor filters [99], optical flow and LBPs [97] or combinations between the two [139].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [99]
Increasing the number of recognized expressions usually follows two main directions: encoding expressions based on FACS action units instead of directly classifying them, and exploiting additional information provided by 3D facial data.
“Increasing the number of recognized expressions usually follows two main directions. First, expressions can be encoded based on FACS AUs [99], [113], [127], [139] instead of directly being classified. This provides a comprehensive coding of FEs without directly making a judgement on their intentionality. Other methods exploit additional information provided by 3D facial data.”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [99]
Cites
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138 · see citing works
Friedman, J., Hastie, T., & Tibshirani, R. (2000) · The Annals of Statistics · doi:10.1214/aos/1016218223 · Cited by 6,884 · see citing works
Ekman, P. (1976) · Medical Entomology and Zoology · Cited by 4,345 · see citing works
A Short Introduction to Boosting
Freund, Y., & Schapire, R. E. (1999) · Cited by 2,991 · see citing works
Comprehensive database for facial expression analysis
Takeo Kanade, A., Cohn, J. F., & Tian, Y. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2000.840611 · Cited by 2,612 · see citing works
Robust real-time object detection.
Viola, P. A. (2001) · Cited by 2,340 · see citing works
Distortion invariant object recognition in the dynamic link architecture
Lades, M. H., Vorbrüggen, J. C., Buhmann, J. M., Lange, J., von der Malsburg, C., Würtz, R. P., & Konen, W. (1993) · IEEE Transactions on Computers · doi:10.1109/12.210173 · Cited by 1,801 · see citing works
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · see citing works
Recognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745 · see citing works
Large Margin DAG's for Multiclass Classification
Platt, J. C., Cristianini, N., & Shawe-Taylor, J. S. (1999) · Cited by 1,637 · see citing works
A brief introduction to boosting
Schapire, R. E. (1999) · Cited by 1,301 · see citing works
Pairwise Classification and Support Vector Machines
Kreßel, U. (1998) · The MIT Press eBooks · doi:10.7551/mitpress/1130.003.0020 · Cited by 1,021 · see citing works
Donato, G., Bartlett, M. S., Hager, J. C., Ekman, P., & B, T. J. S. A. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.799905 · Cited by 999 · see citing works
Self-citationMammone, A., Turchi, M., & Cristianini, N. (2009) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.49 · Cited by 711 · see citing works
EMPATH: A Neural Network that Categorizes Facial Expressions
Dailey, M. N., Cottrell, G. W., Padgett, C. W., & Adolphs, R. (2002) · Journal of Cognitive Neuroscience · doi:10.1162/089892902760807177 · Cited by 235 · see citing works
Classifying facial attributes using a 2-D Gabor wavelet representation and discriminant analysis
Lyons, M. J., Budynek, J., Plante, A., & Akamatsu, S. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2000.840635 · Cited by 162 · see citing works
Cohen, I., Sebe, N., Gozman, F. G., Cirelo, M. C., & Huang, T. S. (2003) · 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. · doi:10.1109/cvpr.2003.1211408 · Cited by 155 · see citing works
Fully automatic upper facial action recognition
Kapoor, A., Qi, Y., & Picard, R. W. (2004) · IEEE International SOI Conference (SOI) · doi:10.1109/amfg.2003.1240843 · Cited by 135 · see citing works
Constraint Classification for Multiclass Classification and Ranking
Har-Peled, S., Roth, D., & Zimak, D. (2002) · Cited by 134 · see citing works
Face Image Analysis by Unsupervised Learning
Bartlett, M. S. (2001) · Springer eBooks · doi:10.1007/978-1-4615-1637-8 · Cited by 109 · see citing works
Self-citationBartlett, M. S., Littlewort, G. C., Fasel, I., Chenu, J., Kanda, T., Ishiguro, H., & Movellan, J. R. (2003) · Cited by 59 · see citing works
Self-citationLearning to Recognize Three-Dimensional Objects
Roth, D., Yang, M., & Ahuja, N. (2002) · Neural Computation · doi:10.1162/089976602753633394 · Cited by 50 · see citing works
Multiclass Learning by Probabilistic Embeddings
Dekel, O., & Singer, Y. (2002) · Cited by 35 · see citing works
Automatic recognition of facial expressions using Bayesian belief networks
Datcu, D., & Rothkrantz, L. J. M. (2005) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2004.1400656 · Cited by 25 · see citing works
CU animate tools for enabling conversations with animated characters
Ma, J., Yan, J., & Cole, R. A. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-119 · Cited by 20 · see citing works
Statistical and neural methods for vision-based analysis of facial expressions and gender
Wilhelm, T., Böhme, H.-J., Gross, H. M., et al. (2005) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2004.1400655 · Cited by 12 · see citing works
3D Tracking of Morphable Objects Using Conditionally Gaussian Nonlinear Filters
Marks, T. K., Hershey, J., Roddey, J. C., et al. (2005) · IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvpr.2004.277 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
Machine Analysis of Facial Expressions
Pantic, M., & Stewart, M. (2007) · Face Recognition · doi:10.5772/4847 · Cited by 248 · see citing works
On deep generative models with applications to recognition
Ranzato, M., Susskind, J., Mnih, V., & Hinton, G. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995710 · Cited by 237 · see citing works
Self-citationAutomatic Facial Expression Recognition Using Facial Animation Parameters and Multistream HMMs
Aleksic, P. S., & Katsaggelos, A. K. (2006) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2005.863510 · Cited by 235 · see citing works
Hierarchical fusion of visual and physiological signals for emotion recognition
Fang, Y., Rong, R., & Huang, J. (2021) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-021-00774-z · Cited by 17 · see citing works
- Authors
- Littlewort, G., Bartlett, M. S., Fasel, I., Susskind, J., & Movellan, J.
- Affiliations
- University of California San Diego (US)
- ORCID
- Littlewort, G.
- Venue
- 2004 Conference on Computer Vision and Pattern Recognition Workshop
- Publisher
- IEEE
- Published
- 2005-04-05
- Pages
- 80-80
- OpenAlex
- W2134393319
- Cited by
- 126
- References
- 27
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 1 · 2022: 3 · 2021: 6 · 2020: 6 · 2019: 1 · 2018: 4 · 2017: 1 · 2016: 7 · 2015: 10 · 2014: 6 · 2013: 7 · 2012: 6
- Type
- conference abstract
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- facial expression recognition, Gabor filters, AdaBoost, support vector machine, linear discriminant analysis, face detection, feature selection, facial action coding, real-time video processing, emotion recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
