The DOI matches its Crossref record; 58 of the 61 references it cites were found in a registry or an index.
Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition
Abstract
Facial expression recognition (FER) is increasingly gaining importance in various emerging affective computing applications. In practice, achieving accurate FER is challenging due to the large amount of inter-personal variations such as expression intensity variations. In this paper, we propose a new spatio-temporal feature representation learning for FER that is robust to expression intensity variations. The proposed method utilizes representative expression-states (e.g., onset, apex and offset of expressions) which can be specified in facial sequences regardless of the expression intensity. The characteristics of facial expressions are encoded in two parts in this paper. As the first part, spatial image characteristics of the representative expression-state frames are learned via a convolutional neural network. Five objective terms are proposed to improve the expression class separability of the spatial feature representation. In the second part, temporal characteristics of the spatial feature representation in the first part are learned with a long short-term memory of the facial expression. Comprehensive experiments have been conducted on a deliberate expression dataset (MMI) and a spontaneous micro-expression dataset (CASME II). Experimental results showed that the proposed method achieved higher recognition rates in both datasets compared to the state-of-the-art methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
(2008) · Journal of Machine Learning Research · Cited by 33,051
Neural networks for pattern recognition
Bishop, C. (1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,537
(2010) · Cited by 14,134
(2010) · International Conference on Machine Learning · Cited by 13,995
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
Face Description with Local Binary Patterns: Application to Face Recognition
Ahonen, T., Hadid, A., & Pietikäinen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.244 · Cited by 5,592
Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846
Flagged first, then by citations
3 of the 61 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- roth, 0, A new 2.5 D representation for lymph node detection using random sets of deep convolutional neural network observations, Proc Medical Image Comput Comput -Assisted Intervention, 520
- chollet, 2015, Keras
Cited by
Sorted by citations
Items 201–256 of 256
Montenegro, J. M. F. (2018) · Research Repository (Kingston University London) · doi:10.48550/arxiv.1810.10941 · Cited by 3
“[149] introduce a similar approach to Chu et al.”
MethodMulti-modal Analysis for the Automatic Evaluation of Epilepsy
Ahmedt‐Aristizabal, D. (2019) · Queensland University of Technology · doi:10.5204/thesis.eprints.132537 · Cited by 2
Weakly Supervised Learning for Facial Behavior Analysis: A ReviewPreprint
Praveen, R. G., Cardinal, P., & Granger, É. (2024) · doi:10.36227/techrxiv.173091765.50249538/v1 · Cited by 2
Detecting Natural Emotions in Virtual Reality Through Facial Movement Analysis
Tare, T., Rahman, R., Choi, H. Y., Lim, J., Lee, G. E., Lee, S.-M., Cho, C.-Y., & Damevski, K. (2025) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/3756884.3766013 · Cited by 1
“Convolutional Neural Networks (CNNs) [17] are widely used to capture spatial features from facial images, while Recurrent Neural Networks (RNNs) [42] help model the changes in facial expressions over time.”
Zhang, Q., Wang, H., Shen, H., Duan, S., Wen, S., & Huang, T. (2025) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2025.3649124 · Cited by 1
Interpretable Facial Relational Network Using Relational Importance.
Kim, S. T., & Ro, Y. M. (2017) · Cited by 1
Self-citationAutomatic Estimation of Enjoyment Levels during Cardiac Rehabilitation Exercise
Wei, H., Moran, K., & O'Connor, N. E. (2018) · International Workshop on Multimedia for Personal Health and Health Care (MMHealth) · doi:10.1145/3264996.3265003 · Cited by 1
Self-Supervised Facial Motion Representation Learning via Contrastive Subclips
Sun, Z., Torrie, S. A., Sumsion, A. W., & Lee, D.-J. (2023) · Electronics · doi:10.3390/electronics12061369 · Cited by 1
“Some later studies [18,19] further proved that the models using the CNN–RNN scheme are practical spatial–temporal feature extractors.”
Kas, M., Ruichek, Y., Merabet, Y. E., & Messoussi, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44137-0_2 · Cited by 1
3DEmo: For Portrait Emotion Recognition with New Dataset
Shao, L., & Agaian, S. С. (2023) · Journal on Computing and Cultural Heritage · doi:10.1145/3631133 · Cited by 2
“Several FER models and datasets have been developed, including ESR-9 [49], EmoFAN [54], CK+ [36], MMI [44], SFEW/AFEW [31], FERPlus [5], and AffectNet [42] for different FER challenges.”
MethodLong Short-Term Perception Network for Dynamic Facial Expression Recognition
Lu, C., Jiang, Y., Fu, K., Zhao, Q., & Yang, H. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8469-5_14 · Cited by 1
Automatic Facial Expression Recognition Employing CNN-Inception and the ResNet 50
Krishnakumari, K., Padmapriya, S., Abirami, N., Kanaghadhara, N. E., & Arul, P. (2023) · International Conference on Research Methodologies in Knowledge Management, Artificial Intelligence and Telecommunication Engineering (RMKMATE) · doi:10.1109/rmkmate59243.2023.10368811 · Cited by 1
“The model in the suggested technique is constructed of symmetric and asymmetric construction pieces, which consist of five convolution-stem layers [20].”
MethodFacial Emotion Recognition (FER) with Deep Learning Algorithm for Sustainable Development
Kumar, A., Sindhwani, M., & Sachdeva, S. (2024) · Green energy and technology · doi:10.1007/978-3-031-47215-2_24 · Cited by 1
Fear Recognition Using Early Biologically Inspired Features Model
Askari, E. (2024) · Journal of Information Systems and Telecommunication (JIST) · doi:10.61186/jist.39051.12.45.12 · Cited by 1
A Comprehensive Investigation on Emotional Detection in Deep Learning
Anand, M., & Babu, S. (2022) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit228111 · Cited by 2
A review of face expression recognition system recommendation for psychiatric disease
Ilham, A., Suryono, S., & Widodo, C. E. (2023) · AIP conference proceedings · doi:10.1063/5.0125595 · Cited by 1
Towards Privacy-Preserving Affect Recognition: A Two-Level Deep Learning ArchitecturePreprint
Mase, J. M., Leesakul, N., Yang, F., Figueredo, G. P., & Torres, M. T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07344 · Cited by 1
Micro‐ and Macro‐Expression Analysis
Allaert, B., Bilasco, I. M., & Djéraba, C. (2022) · Wiley eBooks · doi:10.1002/9781394173853.ch7 · Cited by 1
An Approach Based on Deep Learning for Recognizing Emotion, Gender and AgePreprint
Datta, N., & Sikder, J. (2022) · Research Square · doi:10.21203/rs.3.rs-2367553/v1 · Cited by 1
Facial Emotions Detection using an Efficient Neural Architecture Search NetworkPreprint
Islam, U., Mahum, R., Al‐Salman, A. S., Sharaf, M. A. F., Hassan, H., & Huang, B. (2023) · Preprints.org · doi:10.20944/preprints202309.1273.v1 · Cited by 1
Abdelkawy, H. (2021) · theses.fr (ABES) · doi:10.70675/3ae0f056z1a4az4407z8309zd6153c0e83d7 · Cited by 1
A New Method for Identifying the Relationship of Characters in the Video
Yang, M., & Li, J. (2019) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc47050.2019.9064059
“Similar to this hybrid network, many cascaded networks are used in facial expression recognition (e.g., [14], [15]).”
BackgroundValidation of four designed motion features for recognizing computer user activities
Lee, T., Shen, W.-Y., Lin, R. F., & Cai, Y. (2020) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-020-02479-w
“While previous studies (e.g., Ebrahimi Kahou et al. 2015; Kim et al. 2017) combined RNN and CNN methods to do “end-to-end” modeling, this study applies RNN methods with four handcrafted features to make a computing-cheap methodology.”
Facial Expression Recognition Based on PCD-CNN with Pose and Expression
Dong, H., Xu, J., & Fu, Q. (2020) · Communications in computer and information science · doi:10.1007/978-981-15-7981-3_38
Learning Sequential Patterns Using Spatio-Temporal Attention Networks for Video ClassificationPreprint
Deshmukh, S., Patwardhan, M. S., Mahajan, A., & Deshpande, S. (2021) · Research Square · doi:10.21203/rs.3.rs-148673/v1
“The state of art [17, 18] showed that the architectures of CNN with Long Short-Term Memory (LSTM) can improve recognition performances compared to conventional CNN.”
A Novel Approach of Multi-Objective Based Feature Selection Method for Face Recognition
Gavel, H., & Mehta, A. K. (2021) · IEEE Global Conference for Advancement in Technology (GCAT) · doi:10.1109/gcat52182.2021.9587842
Development of A Real-Time Emotion Recognition System using Facial Expressions
Taneja, & Yogesh, Y. (2021) · International Conference on Recent Developments in Control, Automation & Power Engineering (RDCAPE) · doi:10.1109/rdcape52977.2021.9633406
Emotion Classification from Facial Images and Videos Using a Convolutional Neural Network
(2022) · International Journal of Advanced Trends in Computer Science and Engineering · doi:10.30534/ijatcse/2022/031112022
Rahman, S. M. M., Howlader, T., & Hatzinakos, D. (2019) · Cognitive intelligence and robotics · doi:10.1007/978-981-32-9945-0_4
Allaert, B., Bilasco, I. M., & Djéraba, C. (2022) · doi:10.1002/9781394173853.ch5
Allaert, B., Bilasco, I. M., & Djéraba, C. (2022) · doi:10.1002/9781394173853.ch4
Hierarchical Spatial Pyramidal Feature Representation Learning for Facial Expression Recognition
Gafour, Y., Berrabah, D., Djafri, L., & Benchohra, A. (2022) · International Journal of Organizational and Collective Intelligence · doi:10.4018/ijoci.311094
An Effective Automatic Facial Expression Recognition System Using Deep Neural Networks
Kumar, G. S. N., Reddy, E. V., Dhipti, G. S. N., & Swathi, B. (2023) · Smart innovation, systems and technologies · doi:10.1007/978-981-19-8669-7_60
A Survey on Facial Emotion Recognition
Philip, P., S, A. S., & Zacharias, J. (2023) · International Conference on Control, Communication and Computing (ICCC) · doi:10.1109/iccc57789.2023.10165523
Zhang, P., Zhao, H., Li, M., Chen, Y., Zhang, J., Wang, F., & Wu, X. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394129
“First, human emotion expression is a complex psychological process, and the feature extraction methods such as in [8] and [9] only extract local feature representations of modal data at a fixed scale through a single-scale network model, which is not sufficient to adequately express the complex…”
MethodMultimodal Attention CNN for Human Emotion Recognition
Tiwary, G., Chauhan, S., & Goyal, K. K. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-0641-9_11
Hassan, A. M., Zhou, X., & Aslam, L. (2024) · IEEE Data Driven Control and Learning Systems Conference (DDCLS) · doi:10.1109/ddcls61622.2024.10606687 · Cited by 1
“FER systems can be broadly categorized into two main approaches: static image-based and dynamic sequence-based methods [47].”
BackgroundReview on Facial Emotion Recognition Using Deep Learning With Multiple Databases
Prasad, M. H., Swarnalatha, P., Anantharajah, K., & Prabu, S. (2024) · Advances in medical technologies and clinical practice book series · doi:10.4018/979-8-3693-3719-6.ch018
Analyzing Post-Shopping Facial Expressions: Unraveling Emotions for Enhanced Consumer Insights
Rana, A., Goswami, A., Mazumder, S., Maji, P., Mondal, S. K., & Samanta, R. K. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-75157-8_10
Khabbachi, I., Zouhair, A., & Mahboub, A. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-74491-4_4
Harakannanavar, S. S., Somesh, S., Sughosh, V., & Sumeet, P. (2024) · International Conference on Distributed Systems, Computer Networks and Cybersecurity (ICDSCNC) · doi:10.1109/icdscnc62492.2024.10939593
Face Recognition Using Deep Learning
Kumar, V., Raj, R., & Moudgil, S. (2025) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2025.71935
Object Detection Modeling Using TensorFlow
Rizvi, S. W. A., Priya, R., & Misra, M. K. (2025) · Algorithms for intelligent systems · doi:10.1007/978-981-96-3256-5_26
Video Expression Recognition Method Based on Facial Motion Unit and Temporal Attention
Hu, M. Q., Hu, P., Ge, P., Wang, X., Zhang, K., & Ren, F. (2023) · Journal of Computer-Aided Design & Computer Graphics · doi:10.3724/sp.j.1089.2023.19284
“其中 , DTAGN [11] 70.24 Hasani 等 [20] 77.50 Kim 等 [21] 78.61 Inception-CRF [22] 78.68 Liang 等 [2] 80.71 25.3 MSCNN-PHRNN [12] 81.18 TSG-RNN [3] 81.71 Baddar 等 [23] 81.95 PSRNet [24] 85.23 18.5 本文 87.22 16.9 表 8 9 种方法在 Oulu-CASIA 数据集上准确率对比 方法 准确率 /% 10 − 6 模型参数量 DTAGN [11] 81.46 Baddar 等 [23] 84.58…”
InfluentialResearch on Facial Emotion Recognition Based on Deep Learning
Wang, M. (2025) · Journal of Computing and Electronic Information Management · doi:10.54097/kyr70z32
“In the domain of dynamic facial expression analysis, sequential models such as Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks [12], and Transformer-based architectures have been introduced to capture the temporal evolution of expressions.”
Enhanced Face Emotion Recognition Using CNN and YOLOv8 for Real-Time Applications
Agrawal, S., Bahl, S., Pansari, D., & Tyagi, B. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-95-0684-2_27
Guo, J., Chen, H., Wang, Q., Chen, Y., Cheng, G., Wu, F., & Lim, E. G. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3654065
“Restrictions apply. tasks such as emotion classification and sentiment analysis [20], [21], [22].”
Reinforcement Learning-Inspired Ensemble Framework for Robust Facial Emotion Recognition
Soman, G., & Judy, M. V. (2025) · IEEE International Conference on Green Industrial Electronics and Sustainable Technologies (GIEST) · doi:10.1109/giest66547.2025.11387672
“CNN and LSTM were utilized by authors in [14] to show how facial expressions change across time and space.”
He, J., Zhang, R., Wang, G., Wang, Y., Tang, Y., Lv, J., Shama, A., Cao, J., Ouyang, X., & Liu, G. (2026) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2026.105210
Sumalakshmi, C., & Vasuki, P. (2026) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-026-05093-4
Dynamic Neural Architecture Search for Micro-Expression Recognition
Verma, M., Vipparthi, S. K., Murala, S., & Abdel-Mottaleb, M. S. (2026) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2026.3697628
“Over the years, numerous deep-learning models, including CNNs, graph-based models, and transformers, have been proposed, such as those in [16], [22], [23], [24], [25], and [26].”
A multilayer transformers fusion model with gating network for multimodal emotion recognition
Lv, J., Liang, Y., Chen, H., Gong, X., Li, J., Lin, K., Wang, L., Yuan, J., Zeng, X., & Chen, R. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01209-3
“To address these challenges, Kim et al. [25] designed three networks to extract emotional features from visual and audio modalities, capturing spatiotemporal and dynamic geometric features using a convolutional neural network (CNN)-long short-term memory (LSTM) model.”
Compressed video-driven multimodal modeling and interaction for dynamic expression recognition
Xie, W., Zhang, J., Liang, H., Shen, L., Lai, Z., Song, S., & Yu, Z. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116590
Domain-Category Fusion Guided Diffusion Model for cross-dataset facial expression recognition
Yan, J., Cai, Y., Yue, Y., Wei, J., Hu, J., & Zhou, X. (2026) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2026.104868
Metadata
- Authors
- Kim, D. H., Baddar, W. J., Jang, J., & Ro, Y. M.
- Affiliations
- Korea Advanced Institute of Science and Technology (KR)
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2019-04-01
- Volume
- 10
- Issue
- 2
- Pages
- 223-236
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W2606933083
- Cited by
- 257
- References
- 61
- Citations by year
- 2026: 9 · 2025: 14 · 2024: 33 · 2023: 36 · 2022: 56 · 2021: 39 · 2020: 28 · 2019: 26
- FWCI
- 24.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Affective Computing (publishedVersion)
- KAIST Institutional Repository (KAIST) (submittedVersion)
- Funders
- National Research Foundation of Korea
- Korea government (2015R1A2A2A01005724)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- facial expression recognition, multi-objective learning, long short-term memory, micro-expressions, class separability, affective computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
