CoChat
Sign Up
PreprintCited by 2

Face Expression Recognition and Analysis: The State of the Art

Bettadapura, Vinay·2012·arXiv
DOI 10.48550/arxiv.1203.6722arXiv 1203.6722
Open this paper in CoChat — free account

Abstract

The automatic recognition of facial expressions has been an active research topic since the early nineties. There have been several advances in the past few years in terms of face detection and tracking, feature extraction mechanisms and the techniques used for expression classification. This paper surveys some of the published work since 2001 till date. The paper presents a time-line view of the advances made in this field, the applications of automatic face expression recognizers, the characteristics of an ideal system, the databases that have been used and the advances made in terms of their standardization and a detailed summary of the state of the art. The paper also discusses facial parameterization using FACS Action Units (AUs) and MPEG-4 Facial Animation Parameters (FAPs) and the recent advances in face detection, tracking and feature extraction methods. Notes have also been presented on emotions, expressions and facial features, discussion on the six prototypic expressions and the recent studies on expression classifiers. The paper ends with a note on the challenges and the future work. This paper has been written in a tutorial style with the intention of helping students and researchers who are new to this field.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1203.6722

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Bettadapura (2012) reports accuracies for existing expression recognition software solutions ranging from 55% to 98% since 2001.

    “Bettadapura (2012) reports accuracies for existing expression recognition software solutions ranging from 55% till 98% since 2001.”

    From Towards multimodal emotion recognition in e-learning environments (2016) · cites this paper as Bettadapura (2012)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 61

  1. J. Montagu, "The Expression of the Passions: The Origin and Influence of Charles Le Brun's 'Conférence sur l'expression générale et particulière," Yale University Press, 1994.
  2. C. Darwin, edited by F. Darwin, "The Expression of the Emotions in Man and Animals," 2 nd edition, J. Murray, London, 1904.
  3. Sir C. Bell, "The Anatomy and Philosophy of Expression as Connected with the Fine Arts," 5 th edition, Henry G. Bohn, London, 1865.
  4. A. Samal and P.A. Iyengar, "Automatic Recognition and Analysis of Human Faces and Facial Expressions: A Survey," Pattern Recognition, vol. 25, no. 1, pp. 65-77, 1992.
  5. B. Fasel and J. Luttin, "Automatic Facial Expression Analysis: a survey," Pattern Recognition, vol. 36, no. 1, pp. 259-275, 2003.
  6. Z. Zeng, M. Pantic, G.I. Roisman and T.S. Huang, "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39-58, Jan. 2009.
  7. I. Cohen, N. Sebe, A. Garg, L.S. Chen, and T.S. Huang, "Facial Expression Recognition From Video Sequences: Temporal and Static Modeling", Computer Vision and Image Understanding, vol. 91, pp. 160-187, 2003.
  8. M.S. Bartlett, G. Littlewort, I. Fasel, and R. Movellan, "Real Time Face Detection and Facial Expression Recognition: Development and Application to Human Computer Interaction," Proc. CVPR Workshop on Computer Vision and Pattern Recognition for Human-Computer Interaction, vol. 5, 2003.
  9. P. Michel and R. Kaliouby, "Real Time Facial Expression Recognition in Video Using Support Vector Machines," Proc. 5th Int. Conf. Multimodal Interfaces, Vancouver, BC, Canada, pp. 258-264, 2003.
  10. M. Pantic and J.M. Rothkrantz, "Facial Action Recognition for Facial Expression Analysis from Static Face Images," IEEE Trans. Systems, Man arid Cybernetics Part B, vol. 34, no. 3, pp. 1449-1461, 2004.
  11. I. Buciu and I. Pitas, "Application of Non-Negative and Local Non Negative Matrix Factorization to Facial Expression Recognition," Proc. ICPR, pp. 288- 291, Cambridge, U.K., Aug. 23-26, 2004.
  12. W. Zheng, X. Zhou, C. Zou, and L. Zhao, "Facial Expression Recognition Using Kernel Canonical Correlation Analysis (KCCA)," IEEE Trans. Neural Networks, vol. 17, no. 1, pp. 233-238, Jan. 2006.
  13. K. Anderson and P.W. McOwan, "A Real-Time Automated System for Recognition of Human Facial Expressions," IEEE Trans. Systems, Man, and Cybernetics Part B, vol. 36, no. 1, pp. 96-105, 2006.
  14. M. Pantic and I. Patras, "Dynamics of Facial Expression: Recognition of Facial Actions and Their Temporal Segments Form Face Profile Image Sequences," IEEE Trans. Systems, Man, and Cybernetics Part B, vol. 36, no. 2, pp. 433-449, 2006.
  15. N. Sebe, M.S. Lew, Y. Sun, I. Cohen, T. Gevers, and T.S. Huang, "Authentic Facial Expression Analysis," Image and Vision Computing, vol. 25, pp. 1856- 1863, 2007.
  16. I. Kotsia and I. Pitas, "Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines," IEEE Trans. Image Processing, vol. 16, no. 1, pp. 172-187, 2007.
  17. J. Wang and L. Yin, "Static Topographic Modeling for Facial Expression Recognition and Analysis," Computer Vision and Image Understanding, vol. 108, pp. 19-34, 2007.
  18. F. Dornaika and F. Davoine, "Simultaneous Facial Action Tracking and Expression Recognition in the Presence of Head Motion," Int. J. Computer Vision, vol. 76, no. 3, pp. 257-281, 2008.
  19. I. Kotsia, I. Buciu and I. Pitas, "An Analysis of Facial Expression Recognition under Partial Facial Image Occlusion," Image and Vision Computing, vol. 26, no. 7, pp. 1052-1067, July 2008.
  20. F. Bourel, C.C. Chibelushi and A. A. Low, "Recognition of Facial Expressions in the Presence of Occlusion," Proc. of the Twelfth British Machine Vision Conference, vol. 1, pp. 213-222, 2001.
  21. P. Ekman and W.V. Friesen, "Manual for the Facial Action Coding System," Consulting Psychologists Press, 1977.
  22. J.F. Cohn, Z. Ambadar and P. Ekman, "Observer-Based Measurement of Facial Expression with the Facial Action Coding System", in J.A. Coan and J.B. Allen (Eds.), The handbook of emotion elicitation and assessment, Oxford University Press Series in Affective Science, Oxford, 2005.
  23. I.S. Pandzic and R. Forchheimer (editors), "MPEG-4 Facial Animation: The Standard, Implementation and Applications," John Wiley and sons, 2002.
  24. R. Cowie, E. Douglas-Cowie, K. Karpouzis, G. Caridakis, M. Wallace and S. Kollias, "Recognition of Emotional States in Natural Human-Computer Interaction," in Multimodal User Interfaces, Springer Berlin Heidelberg, 2008.
  25. L. Malatesta, A. Raouzaiou, K. Karpouzis and S. Kollias, "Towards Modeling Embodied Conversational Agent Character Profiles Using Appraisal Theory Predictions in Expression Synthesis," Applied Intelligence, Springer Netherlands, July 2007.
  26. G.A. Abrantes and F. Pereira, "MPEG-4 Facial Animation Technology: Survey, Implementation, and Results," IEEE Trans. Circuits and Systems for Video Technology, vol. 9, no. 2, March 1999.
  27. A. Raouzaiou, N. Tsapatsoulis, K. Karpouzis, and S. Kollias, "Parameterized Facial Expression Synthesis Based on MPEG-4", EURASIP Journal on Applied Signal Processing, vol. 2002, no. 10, pp. 1021-1038, 2002.
  28. P.S. Aleksic, J.J. Williams, Z. Wu and A.K. Katsaggelos, "Audio-Visual Speech Recognition Using MPEG-4 Compliant Visual Features," EURASIP Journal on Applied Signal Processing, vol. 11, pp. 1213-1227, 2002.
  29. M. Pardas and A. Bonafonte, "Facial animation parameters extraction and expression recognition using Hidden Markov Models," Signal Processing: Image Communication, vol. 17, pp. 675-688, 2002.
  30. M.A. Sayette, J.F. Cohn, J.M. Wertz, M.A. Perrott and D.J. Parrott, "A psychometric evaluation of the facial action coding system for assessing spontaneous expression," Journal of Nonverbal Behavior, vol. 25, no. 3, pp. 167-185, Sept. 2001.
  31. W.G. Parrott, "Emotions in Social Psychology," Psychology Press, Philadelphia, October 2000.
  32. P. Ekman and W.V. Friesen, "Constants across cultures in the face and emotions," J. Personality Social Psychology, vol. 17, no. 2, pp. 124-129, 1971.
  33. P. Ekman, "Strong evidence for universals in facial expressions: A reply to Russell's mistaken critique," Psychological Bulletin, vol. 115, no. 2, pp. 268-287, Mar. 1994.
  34. C.E. Izard, "Innate and universal facial expressions: Evidence from developmental and cross-cultural research," Psychological Bulletin, vol. 115, no. 2, pp. 288-299, Mar 1994.
  35. P. Ekman and E.L. Rosenberg, "What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS)", Illustrated Edition, Oxford University Press, 1997.
  36. M. Pantic, I. Patras, "Detecting facial actions and their temporal segments in nearly frontal-view face image sequences," Proc. IEEE conf. Systems, Man and Cybernetics, vol. 4, pp. 3358-3363, Oct. 2005
  37. P. Sinha, "Perceiving and Recognizing Three-Dimensional Forms," Ph.D. dissertation, M. I. T., Cambridge, MA, 1995.
  38. B. Scassellati, "Eye finding via face detection for a foveated, active vision system," in Proc. 15 th Nat. Conf. Artificial Intelligence, pp. 969-976, 1998.
  39. K. Anderson and P.W. McOwan, "Robust real-time face tracker for use in cluttered environments," Computer Vision and Image Understanding, vol. 95, no. 2, pp. 184-200, Aug. 2004.
  40. J. Steffens, E. Elagin, H. Neven, "PersonSpotter-fast and robust system for human detection, tracking and recognition," Proc. 3 rd IEEE Int. Conf. Automatic Face and Gesture Recognition, pp. 516-521, April 1998
  41. I.A. Essa, A.P. Pentland, "Coding, analysis, interpretation, and recognition of facial expressions," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 757-763, July 1997.
  42. P. Viola and M.J. Jones, "Robust real-time object detection," Int. Journal of Computer Vision, vol. 57, no. 2, pp. 137-154, Dec. 2004.
  43. H. Schneiderman, T. Kanade, "Probabilistic modeling of local appearance and spatial relationships for object recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp 45-51, June 1998.
  44. B.D. Lucas and T. Kanade, "An Iterative Image Registration Technique with an Application to Stereo Vision," Proc. 7 th Int. Joint Conf. on Artificial Intelligence (IJCAI), pp. 674-679, Aug. 1981.
  45. N, Sebe, I. Cohen, A. Garg and T.S. Huang, "Machine Learning in Computer Vision," Edition 1, Springer-Verlag, New York, April 2005.
  46. F. Dornaika and J. Ahlberg, "Fast and reliable active appearance model search for 3-D face tracking," IEEE Trans. Systems, Man, and Cybernetics, Part B, vol. 34, no. 4, pp. 1838-1853, Aug. 2004.
  47. M.S. Arulampalam, S. Maskell, N. Gordon and T. Clapp, "A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking," IEEE Trans. Signal Processing, vol. 50, no. 2, pp. 174-188, Feb. 2002.
  48. S. Choi and D. Kim, "Robust Face Tracking Using Motion Prediction in Adaptive Particle Filters," Image Analysis and Recognition, LNCS, vol. 4633, pp. 548- 557, 2007.
  49. F. Xu, J. Cheng and C. Wang, "Real time face tracking using particle filtering and mean shift," Proc. IEEE Int. Conf. Automation and Logistics (ICAL), pp. 2252-2255, Sept. 2008,
  50. H. Wang, S. Chang, "A highly efficient system for automatic face region detection in MPEG video," IEEE Trans. Circuits and Systems for Video Technology, vol. 7, no. 4, pp. 615-628, Aug. 1997.
  51. T. Kanade, J. Cohn, and Y. Tian, "Comprehensive Database for Facial Expression Analysis," Proc. IEEE Int'l Conf. Face and Gesture Recognition (AFGR '00), pp. 46-53, 2000.
  52. L. Yin, X. Wei, Y. Sun, J. Wang, M.J. Rosato, "A 3D facial expression database for facial behavior research," 7 th IEEE Int. Conf. Automatic Face and Gesture Recognition, pp. 211-216, April 2006.
  53. M. Pantic, M.F. Valstar, R. Rademaker, and L. Maat, "Web-Based Database for Facial Expression Analysis," Proc. 13th ACM Int'l Conf. Multimedia (Multimedia '05), pp. 317-321, 2005.
  54. G. Donato, M.S. Bartlett, J.C. Hager, P. Ekman, T.J. Sejnowski, "Classifying facial actions," IEEE. Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 10, pp. 974-989, Oct. 1999.
  55. L.S. Chen, "Joint Processing of Audio-visual Information for the Recognition of Emotional Expressions in Human-computer Interaction," PhD thesis, Univ. of Illinois at Urbana Champaign, Dept. of Elec. Engineering (2000).
  56. M.J. Lyons, S. Akamatsu, M. Kamachi and J. Gyoba, "Coding Facial Expressions with Gabor Wavelets," Proc. 3rd IEEE Int. Conf. on Automatic Face and Gesture Recognition, pp. 200-205, Japan, April 1998.
  57. J. Harrigan, R. Rosenthal, K. Scherer, "New Handbook of Methods in Nonverbal Behavior Research," Oxford University Press, 2008.
  58. F. Bourel, C.C. Chibelushi, and A.A. Low, "Robust Facial Feature Tracking," Proc. 11th British Machine Vision Conference, pp. 232-241, Bristol, England, Sept. 2000.
  59. I. Biederman, "Recognition-by-components: a theory of human image understanding," Psychological Review, vol. 94, no. 2, pp. 115-147, 1987.
  60. M.J. Farah, K.D. Wilson, M. Drain and J. N. Tanaka, "What is 'special' about facial perception?" Psychological Review, vol. 105, no. 3, pp. 482-498, 1998.
  61. P. Ekman, "Telling Lies: Clues to Deceit in the Marketplace, Politics and Marriage," 2 nd Edition, Norton, W.W. and Company, Inc., Sept. 2001.

Cited by

Sorted by citations

Authors
Bettadapura, V.
Venue
arXiv
Publisher
arXiv
Published
2012
DOI
10.48550/arxiv.1203.6722
arXiv
1203.6722
References
5
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review