The DOI matches its Crossref record; 58 of the 61 references it cites were found in a registry or an index.
Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition
Abstract
Facial expression recognition (FER) is increasingly gaining importance in various emerging affective computing applications. In practice, achieving accurate FER is challenging due to the large amount of inter-personal variations such as expression intensity variations. In this paper, we propose a new spatio-temporal feature representation learning for FER that is robust to expression intensity variations. The proposed method utilizes representative expression-states (e.g., onset, apex and offset of expressions) which can be specified in facial sequences regardless of the expression intensity. The characteristics of facial expressions are encoded in two parts in this paper. As the first part, spatial image characteristics of the representative expression-state frames are learned via a convolutional neural network. Five objective terms are proposed to improve the expression class separability of the spatial feature representation. In the second part, temporal characteristics of the spatial feature representation in the first part are learned with a long short-term memory of the facial expression. Comprehensive experiments have been conducted on a deliberate expression dataset (MMI) and a spontaneous micro-expression dataset (CASME II). Experimental results showed that the proposed method achieved higher recognition rates in both datasets compared to the state-of-the-art methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
(2008) · Journal of Machine Learning Research · Cited by 33,051
Neural networks for pattern recognition
Bishop, C. (1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,537
(2010) · Cited by 14,134
(2010) · International Conference on Machine Learning · Cited by 13,995
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
Face Description with Local Binary Patterns: Application to Face Recognition
Ahonen, T., Hadid, A., & Pietikäinen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.244 · Cited by 5,592
Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846
Research works printed but not linked: 2
- roth, 0, A new 2.5 D representation for lymph node detection using random sets of deep convolutional neural network observations, Proc Medical Image Comput Comput -Assisted Intervention, 520
- chollet, 2015, Keras
Cited by
Sorted by citations
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 2,175
“In [71], more intensity states (onset, onset to apex transition, apex, apex to offset transition and offset)”
BackgroundA Brief Review of Facial Emotion Recognition Based on Visual Information
Ko, B. C. (2018) · Sensors · doi:10.3390/s18020401 · Cited by 790
“Therefore, this sub-section briefly introduces some popular databases related to FER consisting of 2D and 3D video sequences and still images:”
BackgroundA systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 713
Facial emotion recognition using deep learning: review and insights
Mellouk, W., & Handouzi, W. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.07.101 · Cited by 370
Zhang, S., Yang, Y., Chen, C., Zhang, X., Leng, Q., & Zhao, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121692 · Cited by 290
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243
Leveraging recent advances in deep learning for audio-Visual emotion recognition
Schoneveld, L., Othmani, A., & Abdelkawy, H. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.03.007 · Cited by 243
“It has been demonstrated that deep neural networks and especially recurrent neural networks are capable of capturing the temporal dynamics of such sequences (Kim et al. [2017]).”
BackgroundZhang, X., Liu, J., Shen, J. G., Li, S., Hou, K., Hu, B., Gao, J. G., Zhang, T., & Hu, B. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2987575 · Cited by 209
Karnati, M., Seal, A., Krejcar, O., & Yazidi, A. (2020) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3031835 · Cited by 198
“D Inception-ResNet + landmarks [56], autoen-coders + SOM [57], spatiotemporal feature + LSTM [58], ensemble DCNNs [59], DCNN for binary classification [60], IACNN [61], 2B(N + M)Softmax [62], S-DSRN [63], color features + Gabor transform [64], DCMA-CNNs [65], broad learning [66], ensemble MLCNNs…”
BackgroundMethodFacial Expression Recognition: A Survey
Huang, Y., Chen, F., Lv, S., & Wang, X. (2019) · Symmetry · doi:10.3390/sym11101189 · Cited by 195
“[96] take advantage of CNN to learn spatial features of representative state frames, and then the LSTM of a facial expression is employed to learn the temporal features of the spatial feature representation.”
MethodFER-net: facial expression recognition using deep neural net
Karnati, M., Seal, A., Krejcar, O., & Yazidi, A. (2021) · Neural Computing and Applications · doi:10.1007/s00521-020-05676-y · Cited by 159
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142
Cai, Y., Li, X., & Li, J. (2023) · Sensors · doi:10.3390/s23052455 · Cited by 149
“Different individuals have different skin colors, looks, and facial features [36,37], which pose challenges to the accuracy of classification.”
BackgroundReddy, S., Karri, S. T., Dubey, S. R., & Mukherjee, S. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852419 · Cited by 119
“a two-step deep learning architecture [32], [35], [36], [37].”
BackgroundExpression snippet transformer for robust video-based facial expression recognition
Liu, Y., Wang, W., Feng, C., Zhang, H., Chen, Z., & Zhan, Y. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109368 · Cited by 124
Recognizing Facial Expressions Using a Shallow Convolutional Neural Network
Miao, S., Xu, H., Han, Z., & Zhu, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2921220 · Cited by 110
Transformer-based Multimodal Information Fusion for Facial Expression Analysis
Zhang, W., Qiu, F., Wang, S., Zeng, H., Zhang, Z., An, R., Ma, B., & Ding, Y. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00271 · Cited by 114
“Some other works [6, 17] utilize LSTM to model the temporal dependencies and characteristics within consecutive facial images.”
STAN: Spatio-Temporal Adversarial Networks for Abnormal Event Detection
Lee, S., Kim, H. G., & Ro, Y. M. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462388 · Cited by 104
Self-citationCogEmoNet: A Cognitive-Feature-Augmented Driver Emotion Recognition Model for Smart Cockpit
Li, W., Zeng, G., Zhang, J., Xu, Y. H., Xing, Y., Zhou, R., Guo, G., Shen, Y., Cao, D., & Wang, F. (2021) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2021.3127935 · Cited by 104
“Cross entropy (CE) [39], F1 [40], mean square error (MSE) [41], and consistency correlation coefficient (CCC) [34] loss function are applied to optimize accuracy (Acc), F1 score, MSE, and CCC respectively.”
InfluentialDeep Joint Spatiotemporal Network (DJSTN) for Efficient Facial Expression Recognition
Jeong, D.-M., Kim, B., & Dong, S.-Y. (2020) · Sensors · doi:10.3390/s20071936 · Cited by 102
“It is even less than STFR model [83] and 3DIR model [11].”
MethodOAENet: Oriented attention ensemble for accurate facial expression recognition
Wang, Z., Zeng, F., Liu, S., & Zeng, B. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107694 · Cited by 105
A comprehensive review of facial expression recognition techniques
Rashmi Adyapady, R., & Annappa, B. (2022) · Multimedia Systems · doi:10.1007/s00530-022-00984-w · Cited by 101
“The FER systems can be divided into two categories, the work considering the static images and those that work with dynamic image sequences [40].”
BackgroundInfluentialMicro-expression recognition: an updated review of current trends, challenges and solutions
Goh, K. M., Ng, C. H., Lim, L. L., & Sheikh, U. U. (2018) · The Visual Computer · doi:10.1007/s00371-018-1607-6 · Cited by 97
“Within high-level approach classification in ME recognition; fully connected layer [80, 81] in CNN is used.”
BackgroundMethodInfluentialMuzammel, M., Salam, H., & Othmani, A. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106433 · Cited by 91
Clip-aware expressive feature learning for video-based facial expression recognition
Liu, Y., Feng, C., Yuan, X., Zhou, L., Wang, W., Qin, J., & Luo, Z. (2022) · Information Sciences · doi:10.1016/j.ins.2022.03.062 · Cited by 83
“…73.23 WMDCNN [42] frame-based 78.2 CER [7] peak frame-based 70.12 AUDN [41] peak frame-based 75.85 LPQ-TOP + SRC [6] sequence-based 64.11 LSTM [13] sequence-based 78.61 DGCN [9] sequence-based 81.53 WMCNN-LSTM [42] sequence-based 87.10 SAANet [43] sequence-based HoloNet [44] frame-based 44.57…”
BackgroundMethodInfluentialFine-Grained Facial Expression Recognition in the Wild
Liang, L., Lang, C., Li, Y., Feng, S., & Zhao, J. C. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.3007327 · Cited by 82
“and CNN+LSTM [32] takes LSTM as temporal network.”
BackgroundFLEPNet: Feature Level Ensemble Parallel Network for Facial Expression Recognition
Karnati, M., Seal, A., Yazidi, A., & Krejcar, O. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3208309 · Cited by 79
“By integrating the properties of facial expressions utilizing DCNN and LSTM, Kim et al. [20] presented a Spatio-temporal feature extraction learning for tackling the FER problem.”
MethodMicro and Macro Facial Expression Recognition Using Advanced Local Motion Patterns
Allaert, B., Bilasco, I. M., & Djéraba, C. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2949559 · Cited by 65
Audio–Visual Fusion for Emotion Recognition in the Valence–Arousal Space Using Joint Cross-Attention
Praveen, R. G., Cardinal, P., & Granger, É. (2023) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2022.3233083 · Cited by 62
“tion with Recurrent Neural Networks (RNN) to capture the effective latent appearance representation, along with temporal dynamics [43].”
MethodEvaluation of the Spatio-Temporal Features and GAN for Micro-Expression Recognition System
Liong, S., Gan, Y. S., Zheng, D., Li, S.-M., Xu, H.-X., Zhang, H., Lyu, R.-K., & Liu, K. (2020) · Journal of Signal Processing Systems · doi:10.1007/s11265-020-01523-4 · Cited by 61
“Due to the emergence of deep learning, the Softmax classifier, normally served as the final fully connected layer, has been employed in the recent works [64, 75, 67, 66].”
BackgroundMethodA Robust Facial Expression Recognition Algorithm Based on Multi-Rate Feature Fusion Scheme
Park, S.-J., Kim, B., & Chilamkurti, N. (2021) · Sensors · doi:10.3390/s21216954 · Cited by 60
“A CNN-RNN or CNN-LSTM [39,40] structure is one of the examples.”
MethodCoReD: Generalizing Fake Media Detection with Continual Representation using Distillation
Kim, M., Tariq, S., & Woo, S. S. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475535 · Cited by 58
Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition
Praveen, R. G., Granger, É., & Cardinal, P. (2021) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg52635.2021.9667055 · Cited by 55
Methods for Facial Expression Recognition with Applications in Challenging Situations
Pise, A. A., Alqahtani, M. A., Verma, P., Purushothama, K., Karras, D. A., Prathibha, S., & Halifa, A. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/9261438 · Cited by 64
“[111] provide a better level of precision than the one that was performed by the Kim group [118].”
MethodOn Driver Behavior Recognition for Increased Safety: A Roadmap
Davoli, L., Martalò, M., Cilfone, A., Belli, L., Ferrari, G., Presta, R., Montanari, R., Mengoni, M., Giraldi, L., Amparore, E. G., et al. (2020) · Safety · doi:10.3390/safety6040055 · Cited by 56
“Hybrid frameworks also exist, in which different architectures’ characteristics are combined together in order to achieve better results [39].”
BackgroundSTCAM: Spatial-Temporal and Channel Attention Module for Dynamic Facial Expression Recognition
Chen, W., Zhang, D., Li, M., & Lee, D.-J. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3027340 · Cited by 55
“Some research focused on modeling the temporal correlation among facial expression frames in the video by modeling temporal correlation separately from spatial correlation [12], [13], [14], [15].”
BackgroundMethodA deep-learning-based facial expression recognition method using textural features
Mukhopadhyay, M., Dey, A., & Kahali, S. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-08005-7 · Cited by 58
Pei, J., & Peng, S. (2019) · Traitement du signal · doi:10.18280/ts.360611 · Cited by 56
Dynamic Objectives Learning for Facial Expression Recognition
Wen, G., Chang, T., Li, H., & Jiang, L. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2966858 · Cited by 53
“A multi-velocity layers are also applied to modify the deep neural network topology for facial expression recognition in video clips [49], while the spatio-temporal feature representation learning is applied to implement expression intensity variations [50].”
Ngo, Q. T., & Yoon, S. (2020) · Sensors · doi:10.3390/s20092639 · Cited by 52
Geometry-Aware Facial Expression Recognition via Attentive Graph Convolutional Networks
Zhao, R., Liu, T., Huang, Z., Lun, D. P.-K., & Lam, K. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3088895 · Cited by 51
“In recent decades, studies [2], [3], [4], [5], [6], [7], [8], [9] on affective computing, pattern recognition, and human-computer interaction have made great efforts to perform automatic recognition and analysis on human emotions from facial images.”
BackgroundSun, X. N., Xia, P., & Ren, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.11.127 · Cited by 49
Compression of dynamic tactile information in the human hand
Shao, Y., Vincent Hayward, B., & Visell, Y. (2020) · Science Advances · doi:10.1126/sciadv.aaz1158 · Cited by 48
Spatial-Temporal Graphs Plus Transformers for Geometry-Guided Facial Expression Recognition
Zhao, R., Liu, T., Huang, Z., Lun, D. P.-K., & Lam, K. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3181736 · Cited by 45
“Kim et al. [26] presented an LSTM framework to learn expression-intensity invariant features through facial sequences for FER.”
BackgroundA Convolutional Neural Network for Compound Micro-Expression Recognition
Zhao, Y., & Xu, J. (2019) · Sensors · doi:10.3390/s19245553 · Cited by 40
E2‐capsule neural networks for facial expression recognition using AU‐aware attention
Cao, S., Yao, Y., & An, G. (2020) · IET Image Processing · doi:10.1049/iet-ipr.2020.0063 · Cited by 39
Jin, X., Lai, Z., & Jin, Z. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3101820 · Cited by 39
Mutual information regularized identity-aware facial expression recognition in compressed video
Liu, X., Jin, L., Han, X., & You, J. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108105 · Cited by 36
An Automatic Recognition Method for Students’ Classroom Behaviors Based on Image Processing
Huang, W., Li, N., Qiu, Z., Jiang, N., Wu, B., & Лю, Б. (2020) · Traitement du signal · doi:10.18280/ts.370318 · Cited by 36
“Kim et al. [18] proposed a method that processes human motions in real time: Based on the information of human motions, the regions of interest (ROIs) were determined in specified parts, and the oriented gradient of each ROI was calculated; After that, human motions were classified by a…”
MethodBaddar, W. J., & Ro, Y. M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013215 · Cited by 6
Self-citationResponsible AI: Gender bias assessment in emotion recognitionPreprint
Domnich, A., & Anbarjafari, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.11436 · Cited by 84
“[78] is another good example of usage a combination of CNN and LSTM.”
BackgroundZhu, X., & Chen, Z. (2019) · The Visual Computer · doi:10.1007/s00371-019-01660-3 · Cited by 35
“Although the spatiotemporal features add dynamic features compared to the static features, they may be negatively affected by the different characteristic of the facial expression (e.g., different transition duration and/or transition type such as onset or offset) [9].”
BackgroundMethodSG-DSN: A Semantic Graph-based Dual-Stream Network for facial expression recognition
Liu, Y. M., Zhang, X., Zhou, J., & Fu, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.017 · Cited by 34
Facial Expression Recognition in the Wild Using Face Graph and Attention
Kim, H.-J., Lee, J.-H., & Ko, B. C. (2023) · IEEE Access · doi:10.1109/access.2023.3286547 · Cited by 32
“In contrast, Jacob and Stenger [12] extracted facial AUs by using an attentionmapwithout applying landmarks.”
BackgroundMethodInfluentialMoorthy, S., & Moon, Y.-K. (2025) · Mathematics · doi:10.3390/math13071100 · Cited by 31
“Kim et al. [52] proposed a frame-based approach to FER, where spatial image features from representative expression-state frames are learned using a convolutional neural network.”
Reducing Videoconferencing Fatigue through Facial Emotion Recognition
Rößler, J., Sun, J., & Gloor, P. A. (2021) · Future Internet · doi:10.3390/fi13050126 · Cited by 32
Adaptive Thresholding of CNN Features for Maize Leaf Disease Classification and Severity Estimation
Mafukidze, H. D., Owomugisha, G., Otim, D., Nechibvute, A., Nyamhere, C., & Mazunga, F. (2022) · Applied Sciences · doi:10.3390/app12178412 · Cited by 36
A Graph-Structured Representation with BRNN for Static-based Facial Expression Recognition
Zhong, L., Bai, C., Li, J., Chen, T., Li, S., & Liu, Y. (2019) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2019.8756615 · Cited by 29
Deep Neural Network Architecture: Application for Facial Expression Recognition
Villanueva, M. G., & Zavala, S. R. (2020) · IEEE Latin America Transactions · doi:10.1109/tla.2020.9099774 · Cited by 29
“Algunos de los estudios representativos que utilizan una combinaci´on de CNN y LSTM (RNN) son: [18], [19], [20], [14], [15], [16], [21], [22].”
BackgroundLi, Q., Zhang, C., Shangguan, W., Wei, Z., Yuan, H., Zhu, J., Li, X., Li, L., Li, G., Liu, P., et al. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122917 · Cited by 29
Tang, J., Li, L., Tang, M., & Xie, J. (2022) · Signal Image and Video Processing · doi:10.1007/s11760-022-02286-0 · Cited by 29
“[13] proposed a new temporal feature learning method.”
BackgroundLARNet: Real-Time Detection of Facial Micro Expression Using Lossless Attention Residual Network
Hashmi, M. F., Ashish, B. K. K., Sharma, V., Keskar, A. G., Bokde, N. D., Yoon, J. H., & Geem, Z. W. (2021) · Sensors · doi:10.3390/s21041098 · Cited by 28
“[34] proposed a model consisting of CNN and long short-term memory (LSTM) to manage spatial and temporal information.”
Verma, M., Lubal, P., Vipparthi, S. K., & Abdel-Mottaleb, M. S. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00475 · Cited by 27
“In the literature, both traditional [18, 6, 37, 17, 5] and CNN-based algorithms [8, 11, 13, 32, 9, 29, 28, 27] have been effective for micro-expression recognition (MER).”
BackgroundNon-Linearities Improve OrigiNet based on Active Imaging for Micro Expression Recognition
Verma, M., Vipparthi, S. K., & Singh, G. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207718 · Cited by 27
“The Spontaneous Actions and Micro-Movements (SAMM) [35] is a newer database recorded at 200 fps and resolution set to 2,040×1,088 pixels.”
BackgroundMethodInfluentialEmotion Recognition Using EEG Signals and Audiovisual Features with Contrastive Learning
Lee, J.-H., Kim, J. Y., & Kim, H. (2024) · Bioengineering · doi:10.3390/bioengineering11100997 · Cited by 25
Li, Z., Zhang, T., Jing, X., & Wang, Y. (2020) · Alexandria Engineering Journal · doi:10.1016/j.aej.2020.10.061 · Cited by 25
Zhao, L., Ge, Y., Guo, S., Li, H., Li, X., Sun, L., & Chen, J. (2024) · Forest Ecology and Management · doi:10.1016/j.foreco.2024.121771 · Cited by 24
Identity-aware Facial Expression Recognition in Compressed Video
Liu, X., Jin, L., Han, X., Lu, J., You, J., & Kong, L. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412820 · Cited by 24
“The recursive neural network (RNN), 3D convolutional, and non-local network can be used to derive information from sequences [3], [4], [5].”
BackgroundMethodInfluentialGajjala, V. R., Reddy, S. P. T., Mukherjee, S., & Dubey, S. R. (2021) · Indian Conference on Computer Vision, Graphics and Image Processing · doi:10.1145/3490035.3490260 · Cited by 26
Efficient neural architecture search for emotion recognition
Verma, M., Mandal, M., Reddy, S. K., Meedimale, Y. R., & Vipparthi, S. K. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119957 · Cited by 24
“Particularly, proposed EmoNAS is 155× and 20× less than the ViT and Squeeze viT Kim et al. (2022) FER models, respectively.”
MethodAttended Relation Feature Representation of Facial Dynamics for Facial Authentication
Kim, S. T., & Ro, Y. M. (2018) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2018.2885276 · Cited by 23
“Recently, several research efforts have been made on facial dynamic encoding for face analysis [18]–[27].”
BackgroundSelf-citationYi, J., Chen, A., Cai, Z., Sima, Y., Zhou, M., & Wu, X. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.105540 · Cited by 23
AutoMER: Spatiotemporal Neural Architecture Search for Microexpression Recognition
Verma, M., Reddy, M. S. K., Meedimale, Y. R., Mandal, M., & Vipparthi, S. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3072290 · Cited by 22
“Recently, the deep learning algorithms based on CNN [1], [2], [16]–[19] have also achieved impressive results in MER.”
BackgroundMethodDeep Learning-Based Holistic Speaker Independent Visual Speech Recognition
Nemani, P., Krishna, G. S., Nikhil, R., Sai, B. D. S., & Kumar, S. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3220190 · Cited by 22
“D Spatio-Temporal Feature Representation: As discussed in Section I, there is a need for the fine-scale representation of the spatio-temporal features of the different video frames [32], [33], [34].”
BackgroundA micro-expression recognition algorithm based on feature enhancement and attention mechanisms
Tang, M., Ling, M., Tang, J., & Hu, J. (2023) · Virtual Reality · doi:10.1007/s10055-023-00808-w · Cited by 22
“The network proposed by Kim et al. (2019) combines CNN and Long Short-Term Memory ( LSTM ).”
MethodEmotion Identification in Movies through Facial Expression Recognition
Almeida, J. E., Vilaça, L., Teixeira, I. N., & Viana, P. (2021) · Applied Sciences · doi:10.3390/app11156827 · Cited by 21
Yang, X., Yan, H., Zhang, A., Xu, P., Pan, S. H., Vai, M. I., & Gao, Y. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105921 · Cited by 23
Zhang, J., Sun, G., Zheng, K., Mazhar, S., Fu, X., Li, Y., & Yu, H. L. (2022) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2022.3163211 · Cited by 20
Human Emotion Recognition in Video Using Subtraction Pre-Processing
He, Z., Tian, J. W., Basu, A., Soraghan, J. J., Caterina, G. D., & Petropoulakis, L. (2019) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3318299.3318321 · Cited by 20
On-the-Fly Facial Expression Prediction Using LSTM Encoded Appearance-Suppressed Dynamics
Baddar, W. J., Lee, S., & Ro, Y. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2957465 · Cited by 22
Self-citationMultimodal emotion recognition using SDA-LDA algorithm in video clips
Tiwari, P., Rathod, H., Thakkar, S., & Darji, A. D. (2021) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-021-03529-7 · Cited by 20
“Based on SAVEE database, Kim et al. (2017) proposed an Informed Segmentation and Labelling Approach (ISLA) that uses speech signals to alter the dynamics of the lower and upper face regions.”
MethodDeep Temporal–Spatial Aggregation for Video-Based Facial Expression Recognition
Pan, X., Guo, W., Guo, X., Li, W., Xu, J., & Wu, J. (2019) · Symmetry · doi:10.3390/sym11010052 · Cited by 19
“The recent deep spatial–temporal networks are designed to recognize facial expression in video and improve recognition performance [39], extracting deep spatial features and temporal features from the video.”
BackgroundHu, M. Q., Ge, P., Wang, X., Lin, H. Z., & Ren, F. (2021) · The Visual Computer · doi:10.1007/s00371-021-02136-z · Cited by 19
“Therefore, some researchers use recurrent neural network (RNN), long short-term memory network (LSTM) or their improved versions [40] to capture temporal information because they perform well in processing sequences with contextual information [41–46].”
BackgroundVAE-Based Adversarial Multimodal Domain Transfer for Video-Level Sentiment Analysis
Wang, Y., Wu, J., Furumai, K., Wada, S., & Kurihara, S. (2022) · IEEE Access · doi:10.1109/access.2022.3174215 · Cited by 19
Dynamic Facial Expression Feature Learning Based on Sparse RNN
Zhi, R., & Wan, M. (2019) · IEEE Joint International Information Technology and Artificial Intelligence Conference (ITAIC) · doi:10.1109/itaic.2019.8785844 · Cited by 20
“Similarly, the characteristics of facial expressions were encoded in two parts in [8].”
BackgroundVariational restricted Boltzmann machines to automated anomaly detection
Demertzis, K., Iliadis, L., Pimenidis, E., & Kikiras, P. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07060-4 · Cited by 18
“Therefore, the solutions are related to the maximization of the Evidence Lower Bound (ELBO), which is equivalent to minimizing the KL deviation, so that [40, 63]:”
MethodViTCN: Hybrid Vision Transformer with Temporal Convolution for Multi-Emotion Recognition
Zakieldin, K., Khattab, R., Ibrahim, E., Arafat, E., Ahmed, N. K., & Hemayed, E. E. (2024) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-024-00436-5 · Cited by 20
Bohi, A., Boudouri, Y. E., & Sfeir, I. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10938-0 · Cited by 23
A multi-scale multi-attention network for dynamic facial expression recognition
Xia, X., Yang, L., Wei, X.-Y., Hichem Sahli, B., & Jiang, D. (2021) · Multimedia Systems · doi:10.1007/s00530-021-00849-8 · Cited by 18
“One common approach is the cascaded framework, where a CNN model is first trained for deep spatial feature learning, followed by a LSTM model for temporal sequence modelling [18].”
MethodRETRACTED ARTICLE: Recognition of emotions in speech using deep CNN and RESNET
Lakshmi, K. L., Muthulakshmi, P., Nithya, A. A., Jeyavathana, R. B., Usharani, R., Das, N. S., & Devi, G. N. R. (2023) · Soft Computing · doi:10.1007/s00500-023-07969-5 · Cited by 18
RetractedDeep 3D Convolutional Neural Network for Facial Micro-Expression Analysis from Video Images
Talluri, K. K., Fiedler, M.-A., & Al-Hamadi, A. K. (2022) · Applied Sciences · doi:10.3390/app122111078 · Cited by 21
“Usually, the spatial information was extracted with CNNs, and subsequently, the features obtained were given into an LSTM to analyze the temporal information [13,14].”
MethodOrtmann, T., Wang, Q., & Putzar, L. (2024) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii63134.2024.00014 · Cited by 16
“While more works focus on static FER, processing single images, dynamic FER systems have demonstrated that including temporal features using multiple frames can be beneficial [11].”
Song Playlist Generator System Based on Facial Expression and Song Mood
Patel, K., & Gupta, R. K. (2021) · International Conference on Artificial Intelligence and Machine Vision (AIMV) · doi:10.1109/aimv53313.2021.9670976 · Cited by 16
FERFM: An Enhanced Facial Emotion Recognition System Using Fine-tuned MobileNetV2 Architecture
Kaur, S., & Kulkarni, N. (2023) · IETE Journal of Research · doi:10.1080/03772063.2023.2202158 · Cited by 19
“[24] CNN spatial features followed by LSTM temporal features”
Human Emotion Recognition Based on Spatio-Temporal Facial Features Using HOG-HOF and VGG-LSTM
Chouhayebi, H., Mahraz, M. A., Riffi, J., Tairi, H., & Alioua, N. (2024) · Computers · doi:10.3390/computers13040101 · Cited by 20
Open-Set Video-based Facial Expression Recognition with Human Expression-sensitive Prompting
Liu, Y., Huang, Y., Liu, S., Zhan, Y., Chen, Z., & Chen, Z. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681583 · Cited by 15
“Currently, many studies use structures like LSTM [19, 22], 3D-CNN [24, 26, 42], Transformer [25, 43] to extract spatio-temporal expression information for V-FER.”
MethodAn Adaptive Optimization Algorithm Based on Hybrid Power and Multidimensional Update Strategy
Hu, J., & Zheng, W. (2019) · IEEE Access · doi:10.1109/access.2019.2897639 · Cited by 15
“A new learning method of spatiotemporal correlation characteristic is proposed to effectively train the model by encoding the spatio-temporal correlation information of the sequence data [20], [21].”
MethodZhen, P., Chen, H., Cheng, Y., Ji, Z., Liu, B., & Yu, H. (2021) · ACM Transactions on Internet of Things · doi:10.1145/3464941 · Cited by 15
“Consequently, LSTM is widely adopted for FER with temporal information [13, 15, 27].”
BackgroundMethodInfluentialEmotion, Age and Gender Recognition using SURF, BRISK, M-SVM and Modified CNN
Sikder, J., Datta, N., Tripura, S., & Das, U. K. (2022) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet55527.2022.9872771 · Cited by 14
Huang, Z., Zhu, Y. M., Li, H., & Yang, D. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108535 · Cited by 15
Metadata
- Authors
- Kim, D. H., Baddar, W. J., Jang, J., & Ro, Y. M.
- Affiliations
- Korea Advanced Institute of Science and Technology (KR)
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2019-04-01
- Volume
- 10
- Issue
- 2
- Pages
- 223-236
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W2606933083
- Cited by
- 257
- References
- 61
- Citations by year
- 2026: 9 · 2025: 14 · 2024: 33 · 2023: 36 · 2022: 56 · 2021: 39 · 2020: 28 · 2019: 26
- FWCI
- 24.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Affective Computing (publishedVersion)
- KAIST Institutional Repository (KAIST) (submittedVersion)
- Funders
- National Research Foundation of Korea
- Korea government (2015R1A2A2A01005724)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- facial expression recognition, multi-objective learning, long short-term memory, micro-expressions, class separability, affective computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
