The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Active and dynamic information fusion for facial expression understanding from image sequences
Abstract
This paper explores the use of multisensory information fusion technique with Dynamic Bayesian networks (DBNs) for modeling and understanding the temporal behaviors of facial expressions in image sequences. Our facial feature detection and tracking based on active IR illumination provides reliable visual information under variable lighting and head motion. Our approach to facial expression recognition lies in the proposed dynamic and probabilistic framework based on combining DBNs with Ekman's Facial Action Coding System (FACS) for systematically modeling the dynamic and stochastic behaviors of spontaneous facial expressions. The framework not only provides a coherent and unified hierarchical probabilistic framework to represent spatial and temporal information related to facial expressions, but also allows us to actively select the most informative visual cues from the available information sources to minimize the ambiguity in recognition. The recognition of facial expressions is accomplished by fusing not only from the current visual observations, but also from the previous visual evidences. Consequently, the recognition becomes more robust and accurate through explicitly modeling temporal behavior of facial expression. In this paper, we present the theoretical foundation underlying the proposed probabilistic and dynamic framework for facial expression modeling and understanding. Experimental results demonstrate that our approach can accurately and robustly recognize spontaneous facial expressions from an image sequence under different conditions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743
(1988) · Cited by 16,582
Eigenfaces vs. Fisherfaces: recognition using class specific linear projection
Belhumeur, P. N., Hespanha, J. P., & Kriegman, D. J. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598228 · Cited by 11,707
Mean shift: a robust approach toward feature space analysis
Comaniciu, D., & Meer, P. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.1000236 · Cited by 11,305
Stochastic models, estimation, and control
(1982) · Mathematics in Science and Engineering · doi:10.1016/s0076-5392(08)x6186-7 · Cited by 3,229
Face recognition by elastic bunch graph matching
Wiskott, L., Fellous, J.-M., Kuiger, N., & von der Malsburg, C. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598235 · Cited by 2,883
Automatic facial expression analysis: a survey
Fasel, B., & Luettin, J. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(02)00052-3 · Cited by 1,810
Face Recognition by Elastic Bunch Graph Matching*†
Wiskott, L., Fellous, J.-M., Krüger, N., & von der Malsburg, C. (2022) · INTELLIGENT BIOMETRIC TECHNIQUES in FINGERPRINT and FACE RECOGNITION · doi:10.1201/9780203750520-11 · Cited by 1,781
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · cited 1×
“A thorough survey of the existing works can be found in [12], [13], [14].”
BackgroundRecognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745
Stochastic models, estimation, and control
(1979) · Mathematics in Science and Engineering · doi:10.1016/s0076-5392(08)x6125-9 · Cited by 1,624
Automatic classification of single facial images
Lyons, M. J., Budynek, J., & Akamatsu, S. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.817413 · Cited by 1,075
Donato, G., Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.799905 · Cited by 999
Anthropometry of the head and face
Farkas, L. G. (1994) · Raven Press eBooks · Cited by 959
Coding, analysis, interpretation, and recognition of facial expressions
Essa, I. A., & Pentland, A. P. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598232 · Cited by 911
Bassili, J. N. (1979) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.37.11.2049 · Cited by 653
Automatic interpretation and coding of face images using flexible models
Lanitis, A., Taylor, C. J., & Cootes, T. F. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598231 · Cited by 625
Zhang, Z., Lyons, M., Schuster, M., & Akamatsu, S. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670990 · Cited by 613
Anthropometry of the head and face
Cacou, C. (1995) · British Journal of Plastic Surgery · doi:10.1016/0007-1226(95)90037-3 · Cited by 585
Analysis and synthesis of facial image sequences using physical and anatomical models
Terzopoulos, D., & Waters, K. (1993) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.216726 · Cited by 538
Recognizing Facial Expressions in Image Sequences Using Local Parameterized Models of Image Motion
Black, M. J., & Yacoob, Y. (1997) · International Journal of Computer Vision · doi:10.1023/a:1007977618277 · Cited by 514
Recognizing human facial expressions from long image sequences using optical flow
Yacoob, Y., & Davis, L. S. (1996) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.506414 · Cited by 458
Facial Action Coding System (FACS)
Ekman, P. (2002) · Medical Entomology and Zoology · Cited by 433
Expert system for automatic analysis of facial expressions
Pantic, M., & Rothkrantz, L. J. M. (2000) · Image and Vision Computing · doi:10.1016/s0262-8856(00)00034-2 · Cited by 415
Facial expressions of emotion: an old controversy and new findings
Ekman, P. (1992) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.1992.0008 · Cited by 407
Recognition of Facial Expression from Optical Flow
Mase, K. (1991) · IEICE Transactions on Information and Systems · Cited by 401
Detection, tracking, and classification of action units in facial expression
Lien, J. J.-J., Kanade, T., Cohn, J. F., & Li, C.-C. (2000) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(99)00103-7 · Cited by 309
Human expression recognition from motion using a radial basis function network architecture
Rosenblum, M., Yacoob, Y., & Davis, L. S. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.536309 · Cited by 284
EMPATH: Face, Emotion, and Gender Recognition Using Holons
Cottrell, G. W., & Metcalfe, J. (1990) · neural information processing systems · Cited by 231
LAFTER: lips and face real time tracker
Oliver, N., Pentland, A. P., & Berard, F. (2002) · Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.1997.609309 · Cited by 199
Feature-point tracking by optical flow discriminates subtle differences in facial expression
Cohn, J. F., Zlochower, A. J., Lien, J. J., & Kanade, T. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670981 · Cited by 181
Representing Face Images for Emotion Classification
Padgett, C. W., & Cottrell, G. W. (1996) · neural information processing systems · Cited by 171
Combining Kalman filtering and mean shift for real time eye tracking under active IR illumination
Zhu, Z., Ji, Q., Fujimura, K., & Lee, K. (2003) · doi:10.1109/icpr.2002.1047460 · Cited by 149
Self-citationHuang, C., & Huang, Y. (1997) · Journal of Visual Communication and Image Representation · doi:10.1006/jvci.1997.0359 · Cited by 137
LAFTER: a real-time face and lips tracker with facial expression recognition
Oliver, N., Pentland, A., & Bérard, F. (2000) · Pattern Recognition · doi:10.1016/s0031-3203(99)00113-2 · Cited by 136
Recognition of Six basic facial expression and their strength by neural network
Kobayashi, H., & Hara, F. (2003) · IEEE RO-MAN · doi:10.1109/roman.1992.253857 · Cited by 116
Bartlett, M. S., Viola, P. A., B, T. J. S. A., Golomb, B. A., Larsen, J. O., Hager, J. C., & Ekman, P. (1995) · Cited by 97
Human emotion recognition from motion using a radial basis function network architecture
Rosenblum, M., Yacoob, Y., & Davis, L. S. (2002) · IEEE Nonrigid and Articulated Motion Workshop · doi:10.1109/mnrao.1994.346256 · Cited by 87
Modeling and Animating Realistic Faces from Images
Pighin, F. P., Szeliski, R., & Salesin, D. (2002) · International Journal of Computer Vision · doi:10.1023/a:1020393915769 · Cited by 81
Magnenat‐Thalmann, N., Kalra, P. K., & Escher, M. (1998) · Proceedings of the IEEE · doi:10.1109/5.664277 · Cited by 59
A probabilistic framework for embedded face and facial expression recognition
Colmenarez, A. J., Frey, B. J., & Huang, T. S. (2003) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1999.786999 · Cited by 41
Statistical cue integration in DAG deformable models
Goldenstein, S., Vogler, C., & Metaxas, D. (2003) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2003.1206510 · Cited by 39
Kearney, G. D., & McKenzie, S. (1993) · Cognitive Science · doi:10.1207/s15516709cog1704_5 · Cited by 36
Zhang, Y., & Ji, Q. (2003) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2003.1238640 · Cited by 36
Self-citationRecognizing facial expressions by spatio-temporal analysis
Yaccob, Y., & Davis, L. (2002) · Proceedings of 12th International Conference on Pattern Recognition · doi:10.1109/icpr.1994.576429 · Cited by 34
Facial expressions of emotion: an old controversy and new findings
Ekman, P. (1992) · doi:10.1093/oso/9780198522614.003.0010 · Cited by 29
Recognition of facial expressions using conceptual fuzzy sets
Ushida, H., Takagi, T., & YAMAGUCHI, T. (2002) · Proceedings of ... IEEE International Conference on Fuzzy Systems · doi:10.1109/fuzzy.1993.327421 · Cited by 28
Neural network approach to component versus holistic recognition of facial expressions in images
Rahardja, A., Sowmya, A., & Wilson, W. H. (1992) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.57047 · Cited by 24
Automatic analysis of of spontaneous facial behavior: A final project report
Bartlett, M. S., Braathen, B., Littlewort-ford, G., Hershey, J., Fasel, I., Marks, T. K., Smith, E., B, T. J. S. A., & Movellan, J. R. (2001) · Cited by 19
Description and Synthesis of Facial Expression Based on Isodensity Maps
Kato, M., So, I., Hishinuma, Y., Nakamura, O., & Minami, T. (1992) · Springer eBooks · doi:10.1007/978-4-431-68204-2_4 · Cited by 19
Research works printed but not linked: 5
- Cottrell, Face, Emotion, Gender Recognition Using Holos, 564
- Zhao, Classifying Facial Emotions by Backpropagation Neural Networks with Fuzzy Inputs, 454
- 10.1037/0022-3514.37.11.2049 · doi:10.1037/0022-3514.37.11.2049 · Paper page
- Tao, Visual Estimation and Compression of Facial Motion Parameters: Elements of a 3D Model-Based Video Coding System, 50, 111
- Morimoto, Framerate Pupil Detector and Gaze Tracker
Cited by
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300
SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer
Ma, J., Tang, L., Fan, F., Huang, J., Mei, X., & Ma, Y. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105686 · Cited by 1,526
Facial Expression Analysis under Partial Occlusion: A Survey
Zhang, L., Verma, B., Tjondronegoro, D., & Chandran, V. (2019) · ACM Computing Surveys · doi:10.1145/3158369 · Cited by 1,258
A 3D Facial Expression Database For Facial Behavior Research
Yin, L., Wei, X., Sun, Y., Wang, J., & Rosato, M. J. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.6 · Cited by 1,171
Metadata
- Authors
- Zhang, Y., & Ji, Q.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2005-05
- Volume
- 27
- Issue
- 5
- Pages
- 699-714
- ISSN
- 0162-8828 (ISSN-L)
- OpenAlex
- W2165183780
- Cited by
- 461
- References
- 51
- FWCI
- 31.51
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.