Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines
Abstract
In this paper, two novel methods for facial expression recognition in facial image sequences are presented. The user has to manually place some of Candide grid nodes to face landmarks depicted at the first frame of the image sequence under examination. The grid-tracking and deformation system used, based on deformable models, tracks the grid in consecutive video frames over time, as the facial expression evolves, until the frame that corresponds to the greatest facial expression intensity. The geometrical displacement of certain selected Candide nodes, defined as the difference of the node coordinates between the first and the greatest facial expression intensity frame, is used as an input to a novel multiclass Support Vector Machine (SVM) system of classifiers that are used to recognize either the six basic facial expressions or a set of chosen Facial Action Units (FAUs). The results on the Cohn-Kanade database show a recognition accuracy of 99.7% for facial expression recognition using the proposed multiclass SVMs and 95.1% for facial expression recognition based on FAU detection.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In later years various ways of dynamic representation were explored.
“In later years various ways of dynamic representation were also explored like tracking geometrical deformations across frames in RGB [114], [116] and 3D [119], [128] or directly extracting features from RGB [127] and thermal frame sequences [196], [210].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [116]
Most early works used geometric representations, like vectors, active contours, or deformable 2D mesh models.
“Most of these early works used geometric representations, like vectors for describing the motion of the face [134], active contours for describing the shape of the mouth and eyebrows [137], or deformable 2D mesh models [116].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [116]
Many methods focus on recognizing a limited set of primary emotions (usually 6), mainly due to a lack of more diverse datasets.
“Many methods focus on recognizing a limited set of primary emotions (usually 6) [115], [116], [123], [130], [137], [145], [146]. This is mainly due to a lack of more diverse datasets.”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [116]
Cites
Wu, Y., & Vapnik, V. N. (1999) · Technometrics · doi:10.2307/1271368 · Cited by 27,236 · see citing works
A Tutorial on Support Vector Machines for Pattern Recognition
Burges, C. J. C. (1998) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009715923555 · Cited by 16,437 · see citing works
Pattern classification and scene analysis
Duda, R. O., & Hart, P. E. (1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895 · see citing works
A comparison of methods for multiclass support vector machines
Hsu, C., & Lin, C. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/72.991427 · Cited by 6,861 · see citing works
An introduction to kernel-based learning algorithms
Müller, K. R., Mika, S., Rätsch, G., Tsuda, K., & Schölkopf, B. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.914517 · Cited by 3,485 · see citing works
Face recognition by elastic bunch graph matching
Wiskott, L., Fellous, J., Kuiger, N., & von der Malsburg, C. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598235 · Cited by 2,888 · see citing works
Support Vector Machines for Classification and Regression
Gunn, S. R. (1998) · Cited by 2,835 · see citing works
Comprehensive database for facial expression analysis
Takeo Kanade, A., Cohn, J. F., & Tian, Y. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2000.840611 · Cited by 2,612 · see citing works
Coding facial expressions with Gabor wavelets
Lyons, M. J., Akamatsu, S., Kamachi, M., & Gyoba, J. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670949 · Cited by 2,214 · see citing works
Automatic facial expression analysis: a survey
Fasel, B., & Luettin, J. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(02)00052-3 · Cited by 1,810 · see citing works
Face Recognition by Elastic Bunch Graph Matching*†
Wiskott, L., Fellous, J., Krüger, N., & von der Malsburg, C. (2022) · Routledge eBooks · doi:10.1201/9780203750520-11 · Cited by 1,781 · see citing works
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · cited 1× · see citing works
“Facial expression recognition A survey on the research made regarding facial expression recognition can be found at [4], [5].”
BackgroundRecognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745 · see citing works
Support vector machines for spam categorization
Drucker, H., Wu, D., & Vapnik, V. N. (1999) · IEEE Transactions on Neural Networks · doi:10.1109/72.788645 · Cited by 1,476 · see citing works
(1972) · Elsevier eBooks · doi:10.1016/c2013-0-02458-9 · Cited by 1,455 · see citing works
Input space versus feature space in kernel-based methods
Schölkopf, B., Mika, S., Burges, C. J. C., Knirsch, P., Müller, K. R., Rätsch, G., & Smola, A. J. (1999) · IEEE Transactions on Neural Networks · doi:10.1109/72.788641 · Cited by 1,159 · see citing works
Automatic classification of single facial images
Lyons, M. J., Budynek, J., & Akamatsu, S. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.817413 · Cited by 1,075 · see citing works
Multiclass Support Vector Machines
(2005) · Springer eBooks · doi:10.1007/1-84628-219-5_3 · Cited by 1,066 · see citing works
Pattern Classification and Scene Analysis.
Grenander, U., Duda, R. O., & Hart, P. E. (1974) · Journal of the American Statistical Association · doi:10.2307/2286028 · Cited by 1,018 · see citing works
Donato, G., Bartlett, M. S., Hager, J. C., Ekman, P., & B, T. J. S. A. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.799905 · Cited by 999 · see citing works
Coding, analysis, interpretation, and recognition of facial expressions
Essa, I. A., & Pentland, A. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598232 · Cited by 906 · see citing works
Facial expression recognition from video sequences: temporal and static modeling
Cohen, I., Sebe, N., Garg, A., Chen, L. S., & Huang, T. S. (2003) · Computer Vision and Image Understanding · doi:10.1016/s1077-3142(03)00081-x · Cited by 874 · see citing works
Support vector machines for 3D object recognition
Pontil, M., & Verri, A. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.683777 · Cited by 815 · see citing works
Matsugu, M., Mori, K., Mitari, Y., & Kaneda, Y. (2003) · Neural Networks · doi:10.1016/s0893-6080(03)00115-1 · Cited by 715 · see citing works
Some statistical issues in the comparison of speech recognition algorithms
Gillick, L., & Cox, S. J. (2003) · International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1989.266481 · Cited by 636 · see citing works
Comparison of classifier methods: a case study in handwritten digit recognition
Bottou, L., Cortes, C., Denker, J. S., Drucker, H., Guyon, I., Jackel, L. D., LeCun, Y., Müller, U., Säckinger, E., Simard, P., et al. (2002) · IAPR International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr.1994.576879 · Cited by 624 · see citing works
Zhang, Z., Lyons, M., Schuster, M., & Akamatsu, S. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670990 · Cited by 613 · see citing works
Bartlett, M. S., Littlewort, G. C., Fasel, I., & Movellan, J. R. (2003) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2003.10057 · Cited by 579 · see citing works
Multiclass Support Vector Machines
Abe, S. (2010) · Advances in computer vision and pattern recognition · doi:10.1007/978-1-84996-098-4_3 · Cited by 512 · see citing works
Active and dynamic information fusion for facial expression understanding from image sequences
Zhang, Y., & Ji, Q. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.93 · Cited by 460 · see citing works
Expert system for automatic analysis of facial expressions
Pantic, M., & Rothkrantz, L. J. M. (2000) · Image and Vision Computing · doi:10.1016/s0262-8856(00)00034-2 · Cited by 415 · see citing works
Real time facial expression recognition in video using support vector machines
Michel, P., & el Kaliouby, R. (2003) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/958432.958479 · Cited by 311 · see citing works
Detection, tracking, and classification of action units in facial expression
Lien, J. J.-J., Takeo Kanade, A., Cohn, J. F., & Li, C.-C. (2000) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(99)00103-7 · Cited by 310 · see citing works
Human expression recognition from motion using a radial basis function network architecture
Rosenblum, M., Yacoob, Y., & Davis, L. S. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.536309 · Cited by 284 · see citing works
Applications of Support Vector Machines to Speech Recognition
Ganapathiraju, A., Hamaker, J., & Picone, J. (2004) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2004.831018 · Cited by 281 · see citing works
Facial expression recognition using a dynamic model and motion energy
Essa, I. A., & Pentland, A. (2002) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.1995.466916 · Cited by 251 · see citing works
Facial Expression Recognition Using Constructive Feedforward Neural Networks
Ma, L., & Khorasani, K. (2004) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2004.825930 · Cited by 247 · see citing works
Automated facial expression recognition based on FACS action units
Lien, J. J., Kanade, T., Cohn, J. F., & Li, C.-C. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670980 · Cited by 226 · see citing works
Tian, Y., Takeo Kanade, A., & Cohn, J. F. (2003) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2002.1004159 · Cited by 224 · see citing works
Tefas, A., Kotropoulos, C. L., & Pitas, I. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.935847 · Cited by 219 · see citing works
Self-citationLearning From Examples in the Small Sample Case: Face Expression Recognition
Guo, G., & Dyer, C. R. (2005) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2005.846658 · Cited by 194 · see citing works
Facial expression recognition: A clustering-based approach
Chen, X., & Huang, T. S. (2003) · Pattern Recognition Letters · doi:10.1016/s0167-8655(02)00371-9 · Cited by 141 · see citing works
Facial expression recognition and synthesis based on an appearance model
Abboud, B., Davoine, F., & Dang, M. (2004) · Signal Processing Image Communication · doi:10.1016/j.image.2004.05.009 · Cited by 134 · see citing works
Facial expression recognition from line-based caricatures
Gao, Y., Leung, M. K. H., Hui, S. C., & Tananda, M. W. (2003) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/tsmca.2003.817057 · Cited by 121 · see citing works
A solution for facial expression representation and recognition
Dubuisson, S., Davoine, F., & Masson, M.-H. (2002) · Signal Processing Image Communication · doi:10.1016/s0923-5965(02)00076-0 · Cited by 117 · see citing works
Real time facial expression recognition in video using support vector machines
Michel, P., & el Kaliouby, R. (2003) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/958468.958479 · Cited by 114 · see citing works
Model-based face tracking for view-independent facial expression recognition
Gokturk, S. B., Bouguet, J.-Y., Tomasi, C., & Girod, B. (2003) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2002.1004168 · Cited by 88 · see citing works
Simultaneous facial action tracking and expression recognition using a particle filter
Dornaika, F., & Davoine, F. (2005) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2005.225 · Cited by 45 · see citing works
FACS [Facial Action Coding System]
Ellgring, J. H. (1987) · Online Publication Service of Würzburg University (Würzburg University) · Cited by 17 · see citing works
Tracking facial features in video sequences using a deformable-model-based approach
Malciu, M., & Preteux, F. J. (2000) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.402450 · Cited by 15 · see citing works
Multiscale Facial Expression Recognition Using Convolutional Neural Networks.
Fasel, B. (2002) · Cited by 14 · see citing works
View and texture-independent facial expression recognition in videos using dynamic programming
Dornaika, F., & Davoine, F. (2005) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2005.1530305 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework
Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.03.001 · Cited by 288 · see citing works
Kim, D. H., Baddar, W. J., Jang, J., & Ro, Y. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2695999 · Cited by 257 · see citing works
Facial Sentiment Analysis Using AI Techniques: State-of-the-Art, Taxonomies, and Challenges
Patel, K., Mehta, D., Mistry, C., Gupta, R., Tanwar, S., Kumar, N., & Alazab, M. (2020) · IEEE Access · doi:10.1109/access.2020.2993803 · Cited by 157 · see citing works
A survey of affect recognition methods: audio, visual and spontaneous expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322216 · Cited by 144 · see citing works
Real-time 2D+3D facial action and expression recognition
Tsalakanidou, F., & Malassiotis, S. (2010) · Pattern Recognition · doi:10.1016/j.patcog.2009.12.009 · Cited by 125 · see citing works
Human Emotion Recognition With Relational Region-Level Analysis
Li, W., Dong, X., & Wang, Y. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3064918 · Cited by 43 · see citing works
- Authors
- Kotsia, I., & Pitas, I.
- Affiliations
- Aristotle University of Thessaloniki (GR)
- ORCID
- Pitas, Ioannis
- Venue
- IEEE Transactions on Image Processing
- Published
- 2007-01
- Volume
- 16
- Issue
- 1
- Pages
- 172-187
- ISSN
- 1057-7149 (ISSN-L)
- OpenAlex
- W2053432263
- Cited by
- 687
- References
- 52
- Citations by year
- 2026: 3 · 2025: 18 · 2024: 26 · 2023: 19 · 2022: 30 · 2021: 40 · 2020: 45 · 2019: 49 · 2018: 40 · 2017: 41 · 2016: 51 · 2015: 46 · 2014: 62 · 2013: 47 · 2012: 43
- FWCI
- 23.98
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Image Processing (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Keywords
- facial expression recognition, facial image sequences, deformable models, facial action units, multi-class support vector machine, facial expression intensity, Cohn-Kanade dataset
- MeSH terms
- Algorithms, Artificial Intelligence, Face, Facial Expression, Humans, Image Enhancement, Image Interpretation, Computer-Assisted, Pattern Recognition, Automated, Subtraction Technique, Video Recording, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
