The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Coding, analysis, interpretation, and recognition of facial expressions
Abstract
We describe a computer vision system for observing facial motion by using an optimal estimation optical flow method coupled with geometric, physical and motion-based dynamic models describing the facial structure. Our method produces a reliable parametric representation of the face's independent muscle action groups, as well as an accurate estimate of facial motion. Previous efforts at analysis of facial expression have been based on the facial action coding system (FACS), a representation developed in order to allow human psychologists to code expression from static pictures. To avoid use of this heuristic coding scheme, we have used our computer vision system to probabilistically characterize facial motion and muscle activation in an experimental population, thus deriving a new, more accurate, representation of human facial expressions that we call FACS+. Finally, we show how this method can be used for coding, analysis, interpretation, and recognition of facial expressions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The expression of the emotions in man and animals.
Darwin, C. (1872) · J. Murray eBooks · doi:10.1037/10001-000 · Cited by 11,657
Bathe, K. (1995) · CERN Document Server (European Organization for Nuclear Research) · Cited by 10,651
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
Finite Element Procedures in Engineering Analysis
Bathe, K. J., & Saunders, H. (1984) · Journal of Pressure Vessel Technology · doi:10.1115/1.3264375 · Cited by 5,223
(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_5794 · Cited by 2,562
View-based and modular eigenspaces for face recognition
Pentland, Moghaddam, & Starner (1994) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition CVPR-94 · doi:10.1109/cvpr.1994.323814 · Cited by 1,800
Mechanical Design in Organisms
Wainwright, S. A., Biggs, W. D., Currey, J. D., & Gosline, J. M. (2020) · Princeton University Press eBooks · doi:10.2307/j.ctv143mdjg · Cited by 1,263
Control System Design: An Introduction to State-Space Methods
Friedland, B., & Stephen, W. (1987) · Cited by 957
Mechanical Design in Organisms
Cromroy, H. L., Wainwright, S. A., Biggs, W. D., Currey, J. D., & Gosline, J. M. (1976) · The Florida Entomologist · doi:10.2307/3494271 · Cited by 899
Bassili, J. N. (1979) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.37.11.2049 · Cited by 653
Research works printed but not linked: 8
- moghaddam, 1994, face recognition using view-based and modular eigenspaces, Automatic Systems for the Identification and Inspection of Humans, 2, 10.1117/12.191877 · doi:10.1117/12.191877
- 10.1145/147156.147180 · doi:10.1145/147156.147180
- 10.1057/palgrave.jibs.8490687 · doi:10.1057/palgrave.jibs.8490687
- bruce, 1988, Recognising Faces
- 10.1002/vis.4340020405 · doi:10.1002/vis.4340020405
- 10.1037/0022-3514.37.11.2049 · doi:10.1037/0022-3514.37.11.2049 · Paper page
- 1993, final report to nsf of the planning workshop on facial expression understanding
- mase, 1991, recognition of facial expressions for optical flow, IEICE Trans Special Issue on Computer Vision and Its Applications, 74
Cited by
The recognition of human movement using temporal templates
Bobick, A. F., & Davis, J. W. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.910878 · Cited by 2,811
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300
Mitra, S., & Acharya, T. (2007) · IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.893280 · Cited by 1,981
Automatic facial expression analysis: a survey
Fasel, B., & Luettin, J. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(02)00052-3 · Cited by 1,810
Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements
Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019) · Psychological Science in the Public Interest · doi:10.1177/1529100619832930 · Cited by 1,748
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778
Recognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745
Action MACH a spatio-temporal Maximum Average Correlation Height filter for action recognition
Rodriguez, M. D., Ahmed, J., & Shah, M. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587727 · Cited by 1,223
A 3D Facial Expression Database For Facial Behavior Research
Yin, L., Wei, X., Sun, Y., Wang, J., & Rosato, M. J. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.6 · Cited by 1,171
Donato, G., Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.799905 · Cited by 999
Multimodal human–computer interaction: A survey
Jaimes, A., & Sebe, N. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2006.10.019 · Cited by 965
Facial expression recognition from video sequences: temporal and static modeling
Cohen, I., Sebe, N., Garg, A., Chen, L. S., & Huang, T. S. (2003) · Computer Vision and Image Understanding · doi:10.1016/s1077-3142(03)00081-x · Cited by 874
Analysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840
Toward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791
Action bank: A high-level representation of activity in video
Sadanand, S., & Corso, J. J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247806 · Cited by 691
Kotsia, I., & Pitas, I. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.884954 · Cited by 687
Zhang, Z., Lyons, M., Schuster, M., & Akamatsu, S. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670990 · Cited by 613
Pantic, M., & Patras, I. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.859075 · Cited by 580
La Cascia, M., Sclaroff, S., & Athitsos, V. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.845375 · Cited by 566
A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition
Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G., & Fu, X. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2485205 · Cited by 520
Active and dynamic information fusion for facial expression understanding from image sequences
Zhang, Y., & Ji, Q. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.93 · Cited by 461
Recognizing human facial expressions from long image sequences using optical flow
Yacoob, Y., & Davis, L. S. (1996) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.506414 · Cited by 458
Measuring facial expressions by computer image analysis
Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999) · Psychophysiology · doi:10.1017/s0048577299971664 · Cited by 424
Facial Action Recognition for Facial Expression Analysis From Static Face Images
Pantic, M., & Rothkrantz, L. J. M. (2004) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2004.825931 · Cited by 417
Expert system for automatic analysis of facial expressions
Pantic, M., & Rothkrantz, L. J. M. (2000) · Image and Vision Computing · doi:10.1016/s0262-8856(00)00034-2 · Cited by 415
Looking at people: sensing for ubiquitous and wearable computing
Pentland, A. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.824823 · Cited by 403
Self-citationGuenter, B., Grimm, C., Wood, D., Malvar, H., & Pighin, F. (2006) · ACM SIGGRAPH 2006 Courses on - SIGGRAPH '06 · doi:10.1145/1185657.1185858 · Cited by 387
A real-time automated system for the recognition of human facial expressions
Anderson, K., & McOwan, P. W. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.854502 · Cited by 355
Tian, Y.-L., Kanade, T., & Cohn, J. F. (2005) · Handbook of Face Recognition · doi:10.1007/0-387-27257-7_12 · Cited by 347
EMOTION IN HUMAN-COMPUTER INTERACTION
(2007) · The Human-Computer Interaction Handbook · doi:10.1201/9781410615862-13 · Cited by 344
Face Expression Recognition and Analysis: The State of the Art
Bettadapura, V. (2012) · arXiv · doi:10.48550/arxiv.1203.6722 · Cited by 336
Automatic determination of facial muscle activations from sparse motion capture marker data
Sifakis, E., Neverov, I., & Fedkiw, R. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073208 · Cited by 312
Real time facial expression recognition in video using support vector machines
Michel, P., & El Kaliouby, R. (2003) · Proceedings of the 5th international conference on Multimodal interfaces · doi:10.1145/958432.958479 · Cited by 308
3D Facial Expression Recognition Based on Primitive Surface Feature Distribution
Wang, J., Yin, L., Wei, X., & Sun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.14 · Cited by 307
Toward multimodal human-computer interface
Sharma, R., Pavlovic, V. I., & Huang, T. S. (1998) · Proceedings of the IEEE · doi:10.1109/5.664275 · Cited by 302
Time–Frequency Representation and Convolutional Neural Network-Based Emotion Recognition
Khare, S. K., & Bajaj, V. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3008938 · Cited by 299
Optical Flow Constraints on Deformable Models with Applications to Face Tracking
DeCarlo, D., & Metaxas, D. (2000) · International Journal of Computer Vision · doi:10.1023/a:1008122917811 · Cited by 299
The First Facial Landmark Tracking in-the-Wild Challenge: Benchmark and Results
Shen, J., Zafeiriou, S., Chrysos, G. G., Kossaifi, J., Tzimiropoulos, G., & Pantic, M. (2015) · 2015 IEEE International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2015.132 · Cited by 292
Bailenson, J. N., Pontikakis, E. D., Mauss, I. B., Gross, J. J., Jabon, M. E., Hutcherson, C. A. C., Nass, C., & John, O. (2008) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.10.011 · Cited by 276
Lopatovska, I., & Arapakis, I. (2011) · Information Processing & Management · doi:10.1016/j.ipm.2010.09.001 · Cited by 263
Guenter, B., Grimm, C., Wood, D., Malvar, H., & Pighin, F. (1998) · Proceedings of the 25th annual conference on Computer graphics and interactive techniques - SIGGRAPH '98 · doi:10.1145/280814.280822 · Cited by 258
Tian, Y., Kanade, T., & Cohn, J. F. (2011) · Handbook of Face Recognition · doi:10.1007/978-0-85729-932-1_19 · Cited by 255
Machine Analysis of Facial Expressions
Pantic, M., & Stewart, M. (2007) · Face Recognition · doi:10.5772/4847 · Cited by 248
Towards an intelligent framework for multimodal affective data analysis
Poria, S., Cambria, E., Hussain, A., & Huang, G.-B. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.10.005 · Cited by 236
(2013) · Cited by 236
Affective state estimation for human–robot Interaction
Kulic, D., & Croft, E. A. (2007) · IEEE Transactions on Robotics · doi:10.1109/tro.2007.904899 · Cited by 230
MULTIMODAL EMOTION RECOGNITION
Sebe, N., Cohen, I., & Huang, T. S. (2005) · Handbook of Pattern Recognition and Computer Vision · doi:10.1142/9789812775320_0021 · Cited by 201
Lisetti, C. L., & Schiano, D. J. (2000) · Pragmatics & Cognition · doi:10.1075/pc.8.1.09lis · Cited by 189
Duric, Z., Gray, W. D., Heishman, R., Li, F., Rosenfeld, A., Schoelles, M. J., Schunn, C., & Wechsler, H. (2002) · Proceedings of the IEEE · doi:10.1109/jproc.2002.801449 · Cited by 185
Multimodal human emotion/expression recognition
Chen, L. S., Huang, T. S., Miyasato, T., & Nakatsu, R. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670976 · Cited by 183
Facial Expression Recognition Using Constructive Feedforward Neural Networks
Ma, L., & Khorasani, K. (2004) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2004.825930 · Cited by 160
ZHANG, Z. (1999) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001499000495 · Cited by 146
Taran, S., & Bajaj, V. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.03.015 · Cited by 110
Spotting segments displaying facial expression from image sequences using HMM
Otsuka, T., & Ohya, J. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670988 · Cited by 69
An expert system for multiple emotional classification of facial expressions
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings 11th International Conference on Tools with Artificial Intelligence · doi:10.1109/tai.1999.809775 · Cited by 48
(2000) · Cited by 27
Metadata
- Authors
- Essa, I. A., & Pentland, A. P.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 1997-07
- Volume
- 19
- Issue
- 7
- Pages
- 757-763
- ISSN
- 0162-8828 (ISSN-L)
- OpenAlex
- W2160947254
- Cited by
- 911
- References
- 34
- FWCI
- 23.78
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.