The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Active and dynamic information fusion for facial expression understanding from image sequences
Abstract
This paper explores the use of multisensory information fusion technique with Dynamic Bayesian networks (DBNs) for modeling and understanding the temporal behaviors of facial expressions in image sequences. Our facial feature detection and tracking based on active IR illumination provides reliable visual information under variable lighting and head motion. Our approach to facial expression recognition lies in the proposed dynamic and probabilistic framework based on combining DBNs with Ekman's Facial Action Coding System (FACS) for systematically modeling the dynamic and stochastic behaviors of spontaneous facial expressions. The framework not only provides a coherent and unified hierarchical probabilistic framework to represent spatial and temporal information related to facial expressions, but also allows us to actively select the most informative visual cues from the available information sources to minimize the ambiguity in recognition. The recognition of facial expressions is accomplished by fusing not only from the current visual observations, but also from the previous visual evidences. Consequently, the recognition becomes more robust and accurate through explicitly modeling temporal behavior of facial expression. In this paper, we present the theoretical foundation underlying the proposed probabilistic and dynamic framework for facial expression modeling and understanding. Experimental results demonstrate that our approach can accurately and robustly recognize spontaneous facial expressions from an image sequence under different conditions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743
(1988) · Cited by 16,582
Eigenfaces vs. Fisherfaces: recognition using class specific linear projection
Belhumeur, P. N., Hespanha, J. P., & Kriegman, D. J. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598228 · Cited by 11,707
Mean shift: a robust approach toward feature space analysis
Comaniciu, D., & Meer, P. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.1000236 · Cited by 11,305
Stochastic models, estimation, and control
(1982) · Mathematics in Science and Engineering · doi:10.1016/s0076-5392(08)x6186-7 · Cited by 3,229
Face recognition by elastic bunch graph matching
Wiskott, L., Fellous, J.-M., Kuiger, N., & von der Malsburg, C. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598235 · Cited by 2,883
Automatic facial expression analysis: a survey
Fasel, B., & Luettin, J. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(02)00052-3 · Cited by 1,810
Face Recognition by Elastic Bunch Graph Matching*†
Wiskott, L., Fellous, J.-M., Krüger, N., & von der Malsburg, C. (2022) · INTELLIGENT BIOMETRIC TECHNIQUES in FINGERPRINT and FACE RECOGNITION · doi:10.1201/9780203750520-11 · Cited by 1,781
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · cited 1×
“A thorough survey of the existing works can be found in [12], [13], [14].”
Background
Research works printed but not linked: 5
- Cottrell, Face, Emotion, Gender Recognition Using Holos, 564
- Zhao, Classifying Facial Emotions by Backpropagation Neural Networks with Fuzzy Inputs, 454
- 10.1037/0022-3514.37.11.2049 · doi:10.1037/0022-3514.37.11.2049 · Paper page
- Tao, Visual Estimation and Compression of Facial Motion Parameters: Elements of a 3D Model-Based Video Coding System, 50, 111
- Morimoto, Framerate Pupil Detector and Gaze Tracker
Cited by
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300
U2Fusion: A Unified Unsupervised Image Fusion Network
Xu, H., Ma, J., Jiang, J., Guo, X., & Ling, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3012548 · Cited by 2,127
SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer
Ma, J., Tang, L., Fan, F., Huang, J., Mei, X., & Ma, Y. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105686 · Cited by 1,526
Facial Expression Analysis under Partial Occlusion: A Survey
Zhang, L., Verma, B., Tjondronegoro, D., & Chandran, V. (2019) · ACM Computing Surveys · doi:10.1145/3158369 · Cited by 1,258
A 3D Facial Expression Database For Facial Behavior Research
Yin, L., Wei, X., Sun, Y., Wang, J., & Rosato, M. J. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.6 · Cited by 1,171
Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism
Li, Y., Zeng, J., Shan, S., & Chen, X. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2886767 · Cited by 916
Social signal processing: Survey of an emerging domain
Vinciarelli, A., Pantic, M., & Bourlard, H. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.11.007 · Cited by 842
Facial expression recognition from near-infrared videos
Zhao, G., Huang, X., Taini, M., Li, S. Z., & Pietikäinen, M. (2011) · Image and Vision Computing · doi:10.1016/j.imavis.2011.07.002 · Cited by 778
(2017) · Cambridge University Press eBooks · doi:10.1017/9781316676202 · Cited by 763
BP4D-Spontaneous: a high-resolution spontaneous 3D dynamic facial expression database
Zhang, X., Yin, L., Cohn, J. F., Canavan, S., Reale, M., Horowitz, A., Liu, P., & Girard, J. M. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.06.002 · Cited by 745
Kotsia, I., & Pitas, I. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.884954 · Cited by 687
Facial Expression Recognition via a Boosted Deep Belief Network
Liu, P., Han, S., Meng, Z., & Tong, Y. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.233 · Cited by 650
Automatic facial expression recognition using features of salient facial patches
Happy, S. L., & Routray, A. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2386334 · Cited by 599
Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships
Tong, Y., Liao, W., & Ji, Q. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1094 · Cited by 378
Self-citationA driver fatigue recognition model based on information fusion and dynamic Bayesian network
Yang, G., Lin, Y., & Bhattacharya, P. (2010) · Information Sciences · doi:10.1016/j.ins.2010.01.011 · Cited by 331
The 2017 IEEE International Conference on Fuzzy Systems
(2016) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2016.2545826 · Cited by 308
Toward Practical Smile Detection
Whitehill, J., Littlewort, G., Fasel, I., Bartlett, M., & Movellan, J. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.42 · Cited by 308
3D Facial Expression Recognition Based on Primitive Surface Feature Distribution
Wang, J., Yin, L., Wei, X., & Sun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.14 · Cited by 307
Fully Automatic Recognition of the Temporal Phases of Facial Actions
Valstar, M. F., & Pantic, M. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2011.2163710 · Cited by 304
Hamm, J., Kohler, C. G., Gur, R. C., & Verma, R. (2011) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2011.06.023 · Cited by 288
Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks
Hasani, B., & Mahoor, M. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.282 · Cited by 287
Machine Analysis of Facial Expressions
Pantic, M., & Stewart, M. (2007) · Face Recognition · doi:10.5772/4847 · Cited by 248
A Dynamic Texture-Based Approach to Recognition of Facial Actions and Their Temporal Models
Koelstra, S., Pantic, M., & Patras, I. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.50 · Cited by 242
Video-based emotion recognition in the wild using deep transfer learning and score fusion
Kaya, H., Gürpınar, F., & Salah, A. A. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.012 · Cited by 235
Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., & Narayanan, S. S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-646 · Cited by 201
Spontaneous vs. posed facial behavior: automatic analysis of brow actions
Valstar, M. F., Pantic, M., Ambadar, Z., & Cohn, J. F. (2006) · Proceedings of the 8th international conference on Multimodal interfaces · doi:10.1145/1180995.1181031 · Cited by 168
A survey of affect recognition methods: audio, visual and spontaneous expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322216 · Cited by 144
Partial least squares regression on grassmannian manifold for emotion recognition
Liu, M., Wang, R., Huang, Z., Shan, S., & Chen, X. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531738 · Cited by 62
Metadata
- Authors
- Zhang, Y., & Ji, Q.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2005-05
- Volume
- 27
- Issue
- 5
- Pages
- 699-714
- ISSN
- 0162-8828 (ISSN-L)
- OpenAlex
- W2165183780
- Cited by
- 461
- References
- 51
- FWCI
- 31.51
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.