How deep neural networks can improve emotion recognition on video data
Abstract
We consider the task of dimensional emotion recognition on video data using deep learning. While several previous methods have shown the benefits of training temporal neural network models such as recurrent neural networks (RNNs) on hand-crafted features, few works have considered combining convolutional neural networks (CNNs) with RNNs. In this work, we present a system that performs emotion recognition on video data using both CNNs and RNNs, and we also analyze how much each neural network component contributes to the system's overall performance. We present our findings on videos from the Audio/Visual+Emotion Challenge (AV+EC2015). In our experiments, we analyze the effects of several hyperparameters on overall performance while also achieving superior performance to the baseline and other competing methods.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Global temporal modeling is found to lead to improved accuracies when compared with simpler methods such as pooling of spatial features.
“Global temporal modeling is found to lead to improved accuracies when compared with simpler methods such as pooling of spatial features (e.g., [152], [90], [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [90]
CNN-LSTM is the more frequent choice among the reviewed studies compared to CNN-RNN.
“We see both CNN-RNN (e.g., [18], [90]) and CNN-LSTM (e.g., [108], [105], [64]) architectures, with CNN-LSTM being the more frequent choice among the reviewed studies.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [90]
Most of these approaches train individual components separately, which may lead to suboptimal performance considering the whole system.
“However, most of these approaches train individual components separately (e.g., [106], [90]), which may lead to suboptimal performance considering the whole system.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [90]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303 · see citing works
Dlib-ml: A Machine Learning Toolkit
King, D. E. (2009) · Journal of machine learning research · doi:10.5555/1577069.1755843 · Cited by 2,945 · see citing works
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300 · see citing works
Evidence for a three-factor theory of emotions
Russell, J. A., & Mehrabian, A. (1977) · Journal of Research in Personality · doi:10.1016/0092-6566(77)90037-x · Cited by 1,940 · see citing works
Universals and cultural differences in the judgments of facial expressions of emotion.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.53.4.712 · Cited by 1,763 · see citing works
Strong evidence for universals in facial expressions: A reply to Russell's mistaken critique.
Ekman, P. (1994) · Psychological Bulletin · doi:10.1037//0033-2909.115.2.268 · Cited by 1,130 · see citing works
Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771 · see citing works
A data-driven approach to cleaning large face datasets
Ng, H., & WINKLER, S. S. (2014) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2014.7025068 · Cited by 668 · see citing works
Facial Expression Recognition via a Boosted Deep Belief Network
Liu, P., Han, S., Meng, Z., & Tong, Y. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.233 · Cited by 667 · see citing works
Combining modality specific deep neural networks for emotion recognition in video
Kahou, S. E., Pal, C., Bouthillier, X., Froumenty, P., Gülçehre, Ç., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R. C., et al. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531745 · Cited by 391 · see citing works
Recurrent Neural Networks for Emotion Recognition in Video
Kahou, S. E., Michalski, V., Konda, K., Memisevic, R., & Pal, C. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2830596 · Cited by 386 · see citing works
Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?
Khorrami, P., Paine, T. L., & Huang, T. S. (2015) · 2015 IEEE International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2015.12 · Cited by 285 · see citing works
Self-citationAU-aware Deep Networks for facial expression recognition
Liu, M., Li, S., Shan, S., & Chen, X. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553734 · Cited by 229 · see citing works
Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data
Ringeval, F., Eyben, F., Kroupi, E., Yuce, A., Thiran, J.-P., Ebrahimi, T., Lalanne, D., & Schuller, B. (2015) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.11.007 · Cited by 202 · cited 1× · see citing works
“While many works [12, 13, 14, 15] have trained recurrent neural networks (RNNs) on hand-crafted features, few works [16, 17] have considered the effects of using a CNN as input to the RNN.”
Background(2015) · doi:10.1145/2808196.2811642 · Cited by 178 · see citing works
He, L., Jiang, D., Yang, L., Pei, E., Wu, P., & Hichem Sahli, B. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811641 · Cited by 149 · see citing works
Long Short Term Memory Recurrent Neural Network based Multimodal Dimensional Emotion Recognition
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811634 · Cited by 124 · see citing works
Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks
Chen, S., & Jin, Q. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811638 · Cited by 111 · see citing works
Finding Celebrities in Billions of Web Images
Zhang, X., Zhang, L., Wang, X., & Shum, H. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2012.2186121 · Cited by 101 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Extended deep neural network for facial emotion recognition
Jain, D. K., Shamsolmoali, P., & Sehdev, P. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.01.008 · Cited by 401 · see citing works
Hybrid deep neural networks for face emotion recognition
Jain, N., Kumar, S., Kumar, A., Shamsolmoali, P., & Zareapoor, M. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.04.010 · Cited by 299 · see citing works
Kim, D. H., Baddar, W. J., Jang, J., & Ro, Y. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2695999 · Cited by 257 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
A Compact Deep Learning Model for Robust Facial Expression Recognition
Kuo, C.-M., Lai, S., & Sarkis, M. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00286 · Cited by 170 · see citing works
Analysing Affective Behavior in the second ABAW2 Competition
Kollias, D., & Zafeiriou, S. P. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00408 · Cited by 168 · see citing works
CNN and LSTM-Based Emotion Charting Using Physiological Signals
Dar, M. N., Akram, M., Khawaja, S. G., & Pujari, A. N. (2020) · Italian National Conference on Sensors · doi:10.3390/s20164551 · Cited by 119 · see citing works
Recognition of Affect in the Wild Using Deep Neural Networks
Kollias, D., Nicolaou, M. A., Kotsia, I., Zhao, G., & Zafeiriou, S. P. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.247 · Cited by 124 · see citing works
Multi-Modal Audio, Video and Physiological Sensor Learning for Continuous Emotion Prediction
Brady, K., Gwon, Y., Khorrami, P., Godoy, E., Campbell, W., Dagli, C., & Huang, T. S. (2016) · Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2988257.2988264 · Cited by 110 · see citing works
Self-citationDeep convolutional BiLSTM fusion network for facial expression recognition
Liang, D., Liang, H., Yu, Z., & Zhang, Y. (2019) · The Visual Computer · doi:10.1007/s00371-019-01636-3 · Cited by 103 · see citing works
Multi-Modal Recurrent Attention Networks for Facial Expression Recognition
Lee, J., Kim, S., Kim, S., & Sohn, K. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2996086 · Cited by 96 · see citing works
Audio-visual emotion recognition using deep transfer learning and multiple temporal models
Ouyang, X., Kawaai, S., Goh, E. G. H., Shen, S., Ding, W., Ming, H., & Huang, D. (2017) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3136755.3143012 · Cited by 92 · see citing works
Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks
Hajj, H. A., Lamard, M., Conze, P.-H., Cochener, B., & Quellec, G. (2018) · Medical Image Analysis · doi:10.1016/j.media.2018.05.001 · Cited by 86 · see citing works
Enhanced speech emotion detection using deep neural networks
Lalitha, S., Tripathi, S., & Gupta, D. (2018) · International Journal of Speech Technology · doi:10.1007/s10772-018-09572-8 · Cited by 82 · see citing works
Santhoshkumar, R., & Geetha, M. K. (2019) · Procedia Computer Science · doi:10.1016/j.procs.2019.05.038 · Cited by 82 · see citing works
Modeling Emotion in Complex Stories: The Stanford Emotional Narratives Dataset
Ong, D. C., Wu, Z., Zhi‐Xuan, T., Reddan, M. C., Kahhalé, I., Mattek, A. M., & Zaki, J. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2955949 · Cited by 77 · see citing works
Learning deep facial expression features from image and optical flow sequences using 3D CNN
Zhao, J., Mao, X., & Zhang, J. (2018) · The Visual Computer · doi:10.1007/s00371-018-1477-y · Cited by 75 · see citing works
Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
Ma, F., Li, Y., Ni, S., Huang, S., & Zhang, L. (2022) · Applied Sciences · doi:10.3390/app12010527 · Cited by 69 · see citing works
Convolutional Neural Networks-An Extensive arena of Deep Learning. A Comprehensive Study
Singh, N., & Sabrol, H. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-021-09551-4 · Cited by 59 · see citing works
Spatio-temporal Pain Recognition in CNN-Based Super-Resolved Facial Images
Bellantonio, M., Haque, M. A., Rodríguez, P., Nasrollahi, K., Telve, T., Escalera, S., Gonzàlez, J., Moeslund, T. B., Rasti, P., & Anbarjafari, G. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-56687-0_13 · Cited by 54 · see citing works
AutoFER: PCA and PSO based automatic facial emotion recognition
Arora, M., & Kumar, M. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09726-4 · Cited by 52 · see citing works
Automatic facial recognition using VGG16 based transfer learning model
Dubey, A. K., & Jain, V. (2020) · Journal of Information and Optimization Sciences · doi:10.1080/02522667.2020.1809126 · Cited by 57 · see citing works
Hybrid-Deep Learning Model for Emotion Recognition Using Facial Expressions
Verma, G., & Verma, H. (2020) · The Review of Socionetwork Strategies · doi:10.1007/s12626-020-00061-6 · Cited by 48 · see citing works
Personalized models for facial emotion recognition through transfer learning
Rescigno, M., Spezialetti, M., & Rossi, S. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09405-4 · Cited by 47 · see citing works
End-to-end learning for dimensional emotion recognition from physiological signals
Keren, G., Kirschstein, T., Marchi, E., Ringeval, F., & Schuller, B. W. (2017) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2017.8019533 · Cited by 45 · see citing works
Tsai, M., & Chen, C.-H. (2021) · IEEE Access · doi:10.1109/access.2021.3052246 · Cited by 41 · see citing works
Du, Z., Wu, S., Huang, D., Li, W., & Wang, Y. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2940224 · Cited by 39 · see citing works
Continuous Emotion Recognition in Videos by Fusing Facial Expression, Head Pose and Eye Gaze
Wu, S., Du, Z., Li, W., Huang, D., & Wang, Y. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3353739 · Cited by 37 · see citing works
Robust emotion recognition from low quality and low bit rate video: A deep learning approach
Cheng, B., Wang, Z., Zhang, Z., Li, Z., Liu, D., Yang, J., Huang, S., & Huang, T. S. (2017) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2017.8273580 · Cited by 32 · see citing works
Self-citationEmotion Classification from Speech and Text in Videos Using a Multimodal Approach
Caschera, M. C., Grifoni, P., & Ferri, F. (2022) · Multimodal Technologies and Interaction · doi:10.3390/mti6040028 · Cited by 34 · see citing works
End-to-End Continuous Emotion Recognition from Video Using 3D Convlstm Networks
Huang, J., Li, Y., Tao, J., Lian, Z., & Yi, J. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461963 · Cited by 32 · see citing works
Two‐Way Feature Extraction Using Sequential and Multimodal Approach for Hateful Meme Classification
Aggarwal, A., Sharma, V., Trivedi, A., Yadav, M. K., Agrawal, C., Singh, D., Mishra, V. K., & Gritli, H. (2021) · Complexity · doi:10.1155/2021/5510253 · Cited by 30 · see citing works
Spatiotemporal Attention Based Deep Neural Networks for Emotion Recognition
Lee, J., Kim, S., Kiim, S., & Sohn, K. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461920 · Cited by 29 · see citing works
Lalitha, S., Gupta, D., Zakariah, M., & Alotaibi, Y. A. (2020) · Applied Acoustics · doi:10.1016/j.apacoust.2020.107519 · Cited by 29 · see citing works
Miyoshi, R., Nagata, N., & Hashimoto, M. (2021) · Neural Computing and Applications · doi:10.1007/s00521-020-05557-4 · Cited by 27 · see citing works
Hu, M. Q., Chu, Q., Wang, X., He, L., & Ren, F. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3063609 · Cited by 27 · see citing works
Sreenivas, V., Namdeo, V., & Kumar, E. V. (2020) · Journal Of Big Data · doi:10.1186/s40537-020-00326-5 · Cited by 24 · see citing works
Kollias, D., & Zafeiriou, S. P. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3014171 · Cited by 23 · see citing works
Dynamic Facial Models for Video-Based Dimensional Affect Estimation
Song, S., Sánchez, E., Tellamekala, M. K., Shen, L., Johnston, A., & Valstar, M. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00200 · Cited by 23 · see citing works
Demochkina, P., & Savchenko, A. V. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68821-9_25 · Cited by 23 · see citing works
A Versatile Multimodal Learning Framework for Zero-Shot Emotion Recognition
Qi, F., Zhang, H., Yang, X., & Xu, C. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3362270 · Cited by 20 · see citing works
On-the-Fly Facial Expression Prediction Using LSTM Encoded Appearance-Suppressed Dynamics
Baddar, W. J., Lee, S., & Ro, Y. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2957465 · Cited by 19 · see citing works
Facial emotion recognition based on deep transfer learning approach
Sultana, A., Dey, S. K., & Rahman, M. A. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15570-z · Cited by 19 · see citing works
A Deep-Learning Based Model for Emotional Evaluation of Video Clips
Kim, B., & Lee, J. (2018) · International Journal of Fuzzy Logic and Intelligent Systems · doi:10.5391/ijfis.2018.18.4.245 · Cited by 19 · see citing works
Attending to Emotional Narratives
Wu, Z., Zhang, X., Zhi‐Xuan, T., Zaki, J., & Ong, D. C. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925497 · Cited by 20 · see citing works
AttendAffectNet–Emotion Prediction of Movie Viewers Using Multimodal Fusion with Self-Attention
Thao, H. T. P., Balamurali, B. T., Roig, G., & Herremans, D. (2021) · Sensors · doi:10.3390/s21248356 · Cited by 16 · see citing works
Enhanced spatio-temporal 3D CNN for facial expression classification in videos
Khanna, D., Jindal, N., Rana, P. S., & Singh, H. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16066-6 · Cited by 16 · see citing works
Mao, Q., Zhu, Q., Rao, Q., Jia, H., & Luo, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916211 · Cited by 16 · see citing works
Activity & Emotion Detection of Recognized kids in CCTV Video for Day Care Using SlowFast & CNN
Darapaneni, N., Singh, D., Chandra, S., Paduri, A. R., Kilhore, N., Chopra, S., Srivastav, A., Tomer, R., & Deshmukh, S. S. (2021) · IEEE World AI IoT Congress (AIIoT) · doi:10.1109/aiiot52608.2021.9454221 · Cited by 15 · see citing works
Rani, S., Bhambri, P., & Chauhan, M. (2021) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait53529.2021.9731203 · Cited by 14 · see citing works
Facial-Expression Recognition from Video using Enhanced Convolutional LSTM
Miyoshi, R., Nagata, N., & Hashimoto, M. (2019) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta47822.2019.8946025 · Cited by 14 · see citing works
Human Emotion Recognition from Body Posture with Machine Learning Techniques
Vaijayanthi, S., & Arunnehru, J. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-12638-3_20 · Cited by 13 · see citing works
Neural style transfer generative adversarial network (NST-GAN) for facial expression recognition
Khemakhem, F., & Ltifi, H. (2023) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-023-00285-6 · Cited by 12 · see citing works
Zero-shot Video Emotion Recognition via Multimodal Protagonist-aware Transformer Network
Qi, F., Yang, X., & Xu, C. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475647 · Cited by 12 · see citing works
Qi, F., Zhang, Z., Yang, X., Zhang, H., & Xu, C. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548278 · Cited by 11 · see citing works
Shen, W., & Li, X. (2020) · International Journal of Intelligent Computing and Cybernetics · doi:10.1108/ijicc-07-2020-0088 · Cited by 10 · see citing works
Izountar, Y., Benbelkacem, S., Otmane, S., Khababa, A., Zenati, N., & Masmoudi, M. (2021) · International Conference on Artificial Intelligence for Cyber Security Systems and Privacy (AI-CSP) · doi:10.1109/ai-csp52968.2021.9671149 · Cited by 9 · see citing works
Facial identity protection using deep learning technologies: an application in affective computing
Mase, J. M., Leesakul, N., Figueredo, G. P., & Torres, M. T. (2022) · AI and Ethics · doi:10.1007/s43681-022-00215-y · Cited by 10 · see citing works
Prediction of Continuous Emotional Measures through Physiological and Visual Data
Joudeh, I. O., Creţu, A.-M., Bouchard, S., & Guimond, S. (2023) · Sensors · doi:10.3390/s23125613 · Cited by 9 · see citing works
Review of Automatic Emotion Recognition Through Facial Expression Analysis
Liliana, D. Y., & Basaruddin, T. (2018) · doi:10.1109/icecos.2018.8605222 · Cited by 9 · see citing works
A Hybrid Facial Expression Recognition System Based on Recurrent Neural Network
Guo, J.-M., Huang, P., & Chang, L.-Y. (2019) · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2019.8909888 · Cited by 9 · see citing works
Facial Expression Recognition by Jointly Partial Image and Deep Metric Learning
Yu, N., & Bai, D. (2019) · IEEE Access · doi:10.1109/access.2019.2963201 · Cited by 9 · see citing works
Zhu, X., Zhang, B., Qiu, Y., & Chepinskiy, S. A. (2024) · IEEE Access · doi:10.1109/access.2024.3358614 · Cited by 9 · see citing works
Joudeh, I. O., Creţu, A.-M., & Bouchard, S. (2024) · Sensors · doi:10.3390/s24134398 · Cited by 7 · see citing works
Salih, W. M., Ibraheem, I. N., & Tariq, A. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-38752-5_19 · Cited by 7 · see citing works
A Survey on Infant Emotion Recognition through Video Clips
Vankudre, G., Ghulaxe, V., Dhomane, A., Badlani, S., & Rane, T. A. (2021) · International Conference on Computational Intelligence and Knowledge Economy (ICCIKE) · doi:10.1109/iccike51210.2021.9410786 · Cited by 6 · see citing works
Fuzzy Emotion Recognition Using Semantic Facial Features and Knowledge-based Fuzzy
Liliana, D. Y., Basaruddin, T., & Widyanto, M. R. (2019) · International Journal of Engineering and Technology · doi:10.21817/ijet/2019/v11i2/191102014 · Cited by 6 · see citing works
Audio-video Emotion Recognition in the Wild using Deep Hybrid NetworksPreprint
Guo, X., Polanía, L. F., & Barner, K. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09023 · Cited by 6 · see citing works
CNN-LSTM-Based Facial Expression Recognition
Lamba, P. S., & Virmani, D. (2021) · Lecture notes in networks and systems · doi:10.1007/978-981-15-9712-1_32 · Cited by 5 · see citing works
A visual self-attention network for facial expression recognition
Yu, N., & Bai, D. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534071 · Cited by 5 · see citing works
Estimation of affective dimensions using CNN-based features of audiovisual data
Basnet, R., Islam, M. T., Howlader, T., Rahman, S. M. M., & Hatzinakos, D. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.09.015 · Cited by 4 · see citing works
Temporal based Emotion Recognition inspired by Activity Recognition models
Mohan, B., & Popa, M. C. (2021) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw52867.2021.9666356 · Cited by 4 · see citing works
Basnet, R., Islam, M. T., Howlader, T., Rahman, S. M. M., & Hatzinakos, D. (2017) · International Conference on New Trends in Computing Sciences (ICTCS) · doi:10.1109/ictcs.2017.14 · Cited by 4 · see citing works
Taking the Hidden Route: Deep Mapping of Affect via 3D Neural Networks
Ceruti, C., Cuculo, V., D’Amelio, A., Grossi, G., & Lanzarotti, R. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-70742-6_18 · Cited by 4 · see citing works
Multimodal Fusion of Spatial-Temporal Features for Emotion Recognition in the Wild
Wang, Z., & Fang, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-77380-3_20 · Cited by 4 · see citing works
Facial Expressions and Body Postures Emotion Recognition based on Convolutional Attention Network
Zhou, T. H., Gao, S., Mei, Y., & Wang, L. (2021) · International Conference on Computer, Information and Telecommunication Systems · doi:10.1109/cits52676.2021.9618520 · Cited by 4 · see citing works
Graph-based Facial Affect Analysis: A Review of Methods, Applications and Challenges.Preprint
Liu, Y. M., Zhang, X., Zhou, J., Li, X., Li, Y., & Zhao, G. (2021) · arXiv (Cornell University) · Cited by 4 · see citing works
Micro-Expression Recognition with Pre-trained Neural Network Models
(2024) · Journal of System and Management Sciences · doi:10.33168/jsms.2024.0604 · Cited by 3 · see citing works
Strizhkova, V., Ferrari, L. M., Kachmar, H., Dantcheva, A., & Brémond, F. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00472 · Cited by 3 · see citing works
Recognizing Emotions from Voice: A Prototype
Rodrigues, M., & Andrade, G. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-38333-5_36 · Cited by 2 · see citing works
Res-RBG Facial Expression Recognition in Image Sequences Based on Dual Neural Networks
Mou, X., Song, Y., Xie, X., You, M., & Wang, R. (2025) · Sensors · doi:10.3390/s25123829 · Cited by 2 · see citing works
Time-Continuous Emotion Recognition Using Spectrogram Based CNN-RNN Modelling
Fedotov, D., Kim, B., Karpov, A. A., & Minker, W. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-26061-3_10 · Cited by 2 · see citing works
Meme Detection For Sentiment Analysis and Human Robot Interactions Using Multiple Modes
Sharma, V., Kushwaha, V., Jaiswal, S., & Nandi, G. C. (2022) · IEEE Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON) · doi:10.1109/upcon56432.2022.9986453 · Cited by 2 · see citing works
Tran, H. N., Phan, P. H., Nguyen, K. H., Hua, H. K., Nguyen, A. Q., Nguyen, H. N. V., & Nguyen, N. V. (2024) · Research Square · doi:10.21203/rs.3.rs-3924396/v1 · Cited by 2 · see citing works
Survey on Emotion Recognition System
Sowmiya, S., & Pamila, J. C. M. J. (2022) · Journal of Information Technology and Digital World · doi:10.36548/jitdw.2022.1.002 · Cited by 1 · see citing works
Exploring Multi-Modality in Animal-Centered Computing
Oren, A. (2021) · International Conference on Animal-Computer Interaction (ACI) · doi:10.1145/3493842.3493900 · Cited by 1 · see citing works
Game Character Facial Animation Using Actor Video Corpus and Recurrent Neural Networks
Schiffer, S. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00113 · Cited by 1 · see citing works
Four-Class Emotion Classification Problem using Deep Learning Classifiers
Zhou, M., Tripathi, A., & Srinivasan, S. (2022) · Journal of Big Data and Artificial Intelligence · doi:10.54116/jbdtp.v1i1.19 · Cited by 1 · see citing works
Movements Recognition in the Human Body Based on Deep Learning Strategies
Mahdi, M. S. (2023) · Journal La Multiapp · doi:10.37899/journallamultiapp.v4i1.773 · Cited by 1 · see citing works
Multimodal Data Fusion and Prediction of Emotional Dimensions Using Deep Neural Network
Bhandari, D., Paul, S., & Narayan, A. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-981-13-1135-2_17 · Cited by 1 · see citing works
ARIĞ, E., & Turan, M. (2020) · European Journal of Science and Technology · doi:10.31590/ejosat.779059 · Cited by 1 · see citing works
Towards Privacy-Preserving Affect Recognition: A Two-Level Deep Learning ArchitecturePreprint
Mase, J. M., Leesakul, N., Yang, F., Figueredo, G. P., & Torres, M. T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07344 · Cited by 1 · see citing works
Survey of Approaches for Building Emotion Detection Applications using a Multi-modal Approach
More, S., Subramanyam, S., Reddy, K., Patil, K., & Chaugule, A. A. (2018) · International Journal of Computer Applications · doi:10.5120/ijca2018916849
Emotion Prediction in Movies Using Visual Features & Genre Information
Aslan, F., & Ekenel, H. K. (2019) · International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk.2019.8907100
Deep Hybrid Neural Networks for Facial Expression Classification
Jadhav, A. B., Burewar, S. L., Waghumbare, A. A., & Gonde, A. B. (2020) · Communications in computer and information science · doi:10.1007/978-981-15-4018-9_26
REVIEW OF AUTOMATIC RECOGNITION METHODS OF HUMAN EMOTIONAL STATE USING IMAGE
Ульянко, А. Л., & Dorofieiev, Y. I. (2020) · Bulletin of National Technical University KhPI Series System Analysis Control and Information Technologies · doi:10.20998/2079-0023.2020.01.15
Emotion Analysis using Facial Expressions in Video
EKMEKÇİ, K., & Özbay, S. (2021) · European Journal of Science and Technology · doi:10.31590/ejosat.926478
Deep Learning Approach for Identifying Emotions in IELTS Speaking Tests
(2019) · International Workshop on Computer Science and Engineering (WCSE) · doi:10.18178/wcse.2019.06.040
Turan, M., & ARIĞ, E. (2021) · European Journal of Science and Technology · doi:10.31590/ejosat.1115837
- Authors
- Khorrami, P., Le Paine, T., Brady, K., Dagli, C., & Huang, T. S.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- MIT Lincoln Laboratory (US)
- Venue
- 2016 IEEE International Conference on Image Processing (ICIP)
- Publisher
- IEEE
- Published
- 2016-09
- Pages
- 619-623
- ISSN
- 1522-4880 (ISSN-L)
- OpenAlex
- W2281407413
- Cited by
- 107
- References
- 20
- Citations by year
- 2025: 4 · 2024: 7 · 2023: 7 · 2022: 8 · 2021: 25 · 2020: 16 · 2019: 19 · 2018: 12 · 2017: 8 · 2016: 1
- FWCI
- 15.42
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - International Conference on Image Processing (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- video-based emotion recognition, convolutional neural networks, recurrent neural networks, hybrid CNN-RNN model, hyperparameter analysis, temporal modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
