The DOI matches its Crossref record; 58 of the 61 references it cites were found in a registry or an index.
Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition
Abstract
Facial expression recognition (FER) is increasingly gaining importance in various emerging affective computing applications. In practice, achieving accurate FER is challenging due to the large amount of inter-personal variations such as expression intensity variations. In this paper, we propose a new spatio-temporal feature representation learning for FER that is robust to expression intensity variations. The proposed method utilizes representative expression-states (e.g., onset, apex and offset of expressions) which can be specified in facial sequences regardless of the expression intensity. The characteristics of facial expressions are encoded in two parts in this paper. As the first part, spatial image characteristics of the representative expression-state frames are learned via a convolutional neural network. Five objective terms are proposed to improve the expression class separability of the spatial feature representation. In the second part, temporal characteristics of the spatial feature representation in the first part are learned with a long short-term memory of the facial expression. Comprehensive experiments have been conducted on a deliberate expression dataset (MMI) and a spontaneous micro-expression dataset (CASME II). Experimental results showed that the proposed method achieved higher recognition rates in both datasets compared to the state-of-the-art methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
(2008) · Journal of Machine Learning Research · Cited by 33,051
Neural networks for pattern recognition
Bishop, C. (1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,537
(2010) · Cited by 14,134
(2010) · International Conference on Machine Learning · Cited by 13,995
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
Face Description with Local Binary Patterns: Application to Face Recognition
Ahonen, T., Hadid, A., & Pietikäinen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.244 · Cited by 5,592
Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299101 · Cited by 2,152
Ekman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,971
Face recognition by independent component analysis
Bartlett, M. S., Movellan, J. R., & B, T. J. S. A. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.804287 · Cited by 1,951
Automatic facial expression analysis: a survey
Fasel, B., & Luettin, J. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(02)00052-3 · Cited by 1,810
Going deeper in facial expression recognition using deep neural networks
Mollahosseini, A., Chan, D., & Mahoor, M. H. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477450 · Cited by 1,155
Fast Normalized Cross-Correlation
Yoo, J.-C., & Han, T. H. (2009) · Circuits Systems and Signal Processing · doi:10.1007/s00034-009-9130-7 · Cited by 1,128
Web-Based Database for Facial Expression Analysis
Pantić, M., Valstar, M., Rademaker, R. M., & Maat, L. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521424 · Cited by 1,124
Donato, G., Bartlett, M. S., Hager, J. C., Ekman, P., & B, T. J. S. A. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.799905 · Cited by 999
CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation
Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., & Fu, X. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086041 · Cited by 993
Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition
Jung, H., Lee, S., Yim, J., Park, S., & Kim, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.341 · Cited by 781 · cited 6×
“In [32], a temporal geometric feature was jointly learned in order to reduce the effect of the identity on the learned spatio-temporal appearance features.”
BackgroundMethodInfluentialKotsia, I., & Pitas, I. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.884954 · Cited by 687
Action Recognition using Visual AttentionPreprint
Sharma, S., Kiros, R., & Salakhutdinov, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04119 · Cited by 623
PCCA: A new approach for distance learning from sparse pairwise constraints
Mignon, A., & Jurie, F. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247987 · Cited by 621
Deep Feature Learning for Knee Cartilage Segmentation Using a Triplanar Convolutional Neural Network
Prasoon, A., Petersen, K., Igel, C., Lauze, F., Dam, E., & Nielsen, M. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_31 · Cited by 582
The computer expression recognition toolbox (CERT)
Littlewort, G. C., Whitehill, J., Wu, T., Fasel, I., Frank, M. G., Movellan, J. R., & Bartlett, M. S. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771414 · Cited by 571
Local Directional Number Pattern for Face Analysis: Face and Expression Recognition
Rivera, A. R., Castillo, J. R., & Chae, O. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2235848 · Cited by 514
Incremental Face Alignment in the Wild
Asthana, A., Zafeiriou, S. P., Cheng, S., & Pantić, M. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.240 · Cited by 453
Notes on Convolutional Neural NetworksPreprint
Bouvrie, J. (2006) · CogPrints (University of Southampton) · Cited by 435
Tian, Y., Takeo Kanade, A., & Cohn, J. F. (2005) · Springer eBooks · doi:10.1007/0-387-27257-7_12 · Cited by 407
Recurrent Neural Networks for Emotion Recognition in Video
Kahou, S. E., Michalski, V., Konda, K., Memisevic, R., & Pal, C. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2830596 · Cited by 396
Fully Automatic Facial Action Unit Detection and Temporal Analysis
Valstar, M., & Pantić, M. (2006) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2006.85 · Cited by 352
A real-time automated system for the recognition of human facial expressions
Anderson, K., & McOwan, P. W. (2006) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2005.854502 · Cited by 351
Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis
Liu, M., Li, S., Shan, S., Wang, R., & Chen, X. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16817-3_10 · Cited by 333
Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?
Khorrami, P., Paine, T. L., & Huang, T. S. (2015) · 2015 IEEE International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2015.12 · Cited by 285
Fast normalized cross correlation for defect detection
Tsai, D., & Lin, C.-T. (2003) · Pattern Recognition Letters · doi:10.1016/s0167-8655(03)00106-5 · Cited by 281
Gabor wavelets and General Discriminant Analysis for face identification and verification
Shen, L., Bai, L., & Fairhurst, M. C. (2006) · Image and Vision Computing · doi:10.1016/j.imavis.2006.05.002 · Cited by 266
Action unit detection using sparse appearance descriptors in space-time video volumes
Jiang, B., Valstar, M., & Pantić, M. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771416 · Cited by 236
Cross-entropy vs. squared error training: a theoretical and experimental comparison
Golik, P., Doetsch, P., & Ney, H. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-436 · Cited by 231
AU-aware Deep Networks for facial expression recognition
Liu, M., Li, S., Shan, S., & Chen, X. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553734 · Cited by 229
AU-inspired Deep Networks for Facial Expression Feature Learning
Liu, M., Li, S., Shan, S., & Chen, X. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.02.011 · Cited by 223
Evaluation of Face Resolution for Expression Analysis
Tian, Y. (2005) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvpr.2004.334 · Cited by 207
Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression Recognition
Wang, Y., See, J., Phan, R. C. ‐., & Oh, Y.-H. (2015) · PLoS ONE · doi:10.1371/journal.pone.0124674 · Cited by 146
A Dynamic Appearance Descriptor Approach to Facial Actions Temporal Modeling
Jiang, B., Valstar, M., Martínez, B., & Pantić, M. (2013) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2013.2249063 · Cited by 146
Long Short Term Memory Recurrent Neural Network based Multimodal Dimensional Emotion Recognition
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811634 · Cited by 124
Lee, S. H., Plataniotis, K. N., & Ro, Y. M. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2346515 · Cited by 124
Self-citationHow deep neural networks can improve emotion recognition on video data
Khorrami, P., Le Paine, T., Brady, K., Dagli, C., & Huang, T. S. (2016) · 2016 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2016.7532431 · Cited by 107
Robust Facial Expression Recognition via Compressive Sensing
Zhang, S., Zhao, X., & Lei, B. (2012) · Sensors · doi:10.3390/s120303747 · Cited by 75
A new method for facial expression recognition based on sparse representation plus LBP
Huang, M., Wang, Z., & Ying, Z. (2010) · International Congress on Image and Signal Processing (CISP) · doi:10.1109/cisp.2010.5647898 · Cited by 74
Sparse representations for facial expressions recognition via l1 optimization
Zafeiriou, S. P., & Petrou, M. (2010) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2010.5543148 · Cited by 73
Facial Expression Recognition Based on Local Phase Quantization and Sparse Representation
Wang, Z., & Ying, Z. (2012) · International Conference on Natural Computation (ICNC) · doi:10.1109/icnc.2012.6234551 · Cited by 70
Quantifying Micro-expressions with Constraint Local Model and Local Binary Pattern
Yan, W., Wang, S., Chen, Y.-H., Zhao, G., & Fu, X. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_20 · Cited by 59
Chan, C. H., Kittler, J., Poh, N., Ahonen, T., & Pietikäinen, M. (2009) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2009.5457642 · Cited by 58
Monogenic Riesz wavelet representation for micro-expression recognition
Oh, Y.-H., Ngo, A. C. L., See, J., Liong, S., Phan, R. C. ‐., & Ling, H.-C. (2015) · IEEE International Conference on Digital Signal Processing (DSP) · doi:10.1109/icdsp.2015.7252078 · Cited by 57
Lee, S. H., Baddar, W. J., & Ro, Y. M. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2015.12.016 · Cited by 54
Self-citationSubtle Facial Expression Recognition Using Adaptive Magnification of Discriminative Facial Motion
Park, S., Lee, S. H., & Ro, Y. M. (2015) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2733373.2806362 · Cited by 41
Self-citationFacial expression recognition using geometric and appearance features
Chen, J., Chen, D., Gong, Y., Meng, Y., Zhang, K., & Wang, L. (2012) · International Conference on Internet Multimedia Computing and Service · doi:10.1145/2382336.2382345 · Cited by 39
Lee, S. H., & Ro, Y. M. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2496320 · Cited by 31
Self-citationAutomatic Facial Expression Recognition Using Extended AR-LBP
Naika, C. L. S., Jha, S. S., Das, P. K., & Nair, S. B. (2012) · Communications in computer and information science · doi:10.1007/978-3-642-31686-9_29 · Cited by 8
Research works printed but not linked: 2
- roth, 0, A new 2.5 D representation for lymph node detection using random sets of deep convolutional neural network observations, Proc Medical Image Comput Comput -Assisted Intervention, 520
- chollet, 2015, Keras
Cited by
Sorted by citations
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 2,175
“In [71], more intensity states (onset, onset to apex transition, apex, apex to offset transition and offset)”
BackgroundA Brief Review of Facial Emotion Recognition Based on Visual Information
Ko, B. C. (2018) · Sensors · doi:10.3390/s18020401 · Cited by 790
“Therefore, this sub-section briefly introduces some popular databases related to FER consisting of 2D and 3D video sequences and still images:”
BackgroundA systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 713
Facial emotion recognition using deep learning: review and insights
Mellouk, W., & Handouzi, W. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.07.101 · Cited by 370
Zhang, S., Yang, Y., Chen, C., Zhang, X., Leng, Q., & Zhao, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121692 · Cited by 290
Metadata
- Authors
- Kim, D. H., Baddar, W. J., Jang, J., & Ro, Y. M.
- Affiliations
- Korea Advanced Institute of Science and Technology (KR)
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2019-04-01
- Volume
- 10
- Issue
- 2
- Pages
- 223-236
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W2606933083
- Cited by
- 257
- References
- 61
- Citations by year
- 2026: 9 · 2025: 14 · 2024: 33 · 2023: 36 · 2022: 56 · 2021: 39 · 2020: 28 · 2019: 26
- FWCI
- 24.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Affective Computing (publishedVersion)
- KAIST Institutional Repository (KAIST) (submittedVersion)
- Funders
- National Research Foundation of Korea
- Korea government (2015R1A2A2A01005724)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- facial expression recognition, multi-objective learning, long short-term memory, micro-expressions, class separability, affective computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
