The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
A Brief Review of Facial Emotion Recognition Based on Visual Information
Open accessAbstract
Facial emotion recognition (FER) is an important topic in the fields of computer vision and artificial intelligence owing to its significant academic and commercial potential. Although FER can be conducted using multiple sensors, this review focuses on studies that exclusively use facial images, because visual expressions are one of the main information channels in interpersonal communication. This paper provides a brief review of researches in the field of FER conducted over the past decades. First, conventional FER approaches are described along with a summary of the representative categories of FER systems and their main algorithms. Deep-learning-based FER approaches using deep networks enabling “end-to-end” learning are then presented. This review also focuses on an up-to-date hybrid deep-learning approach combining a convolutional neural network (CNN) for the spatial features of an individual frame and long short-term memory (LSTM) for temporal features of consecutive frames. In the later part of this paper, a brief review of publicly available evaluation metrics is given, and a comparison with benchmark results, which are a standard for a quantitative comparison of FER researches, is described. This review can serve as a brief guidebook to newcomers in the field of FER, providing basic knowledge and a general understanding of the latest state-of-the-art studies, as well as to experienced researchers looking for productive directions for future work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303
Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846
Long-Term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2599174 · Cited by 1,547
A 3D Facial Expression Database For Facial Behavior Research
Yin, L., Wei, X., Sun, Y., Wang, J., & Rosato, M. J. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.6 · Cited by 1,171
Going deeper in facial expression recognition using deep neural networks
Mollahosseini, A., Chan, D., & Mahoor, M. H. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477450 · Cited by 1,086
CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation
Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., & Fu, X. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086041 · Cited by 993
Compound facial expressions of emotion
Du, S., Tao, Y., & Martinez, A. M. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1322355111 · Cited by 806
DISFA: A Spontaneous Facial Action Intensity Database
Mavadati, S. M., Mahoor, M. H., Bartlett, K., Trinh, P., & Cohn, J. F. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2013.4 · Cited by 793
Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition
Jung, H., Lee, S., Yim, J., Park, S., & Kim, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.341 · Cited by 781 · cited 3×
“• Joint fine-tuning in deep neural networks [48] Sequence 70.”
BackgroundMethodInfluentialFacial expression recognition from near-infrared videos
Zhao, G., Huang, X., Taini, M., Li, S. Z., & Pietikäinen, M. (2011) · Image and Vision Computing · doi:10.1016/j.imavis.2011.07.002 · Cited by 778
BP4D-Spontaneous: a high-resolution spontaneous 3D dynamic facial expression database
Zhang, X., Yin, L., Cohn, J. F., Canavan, S., Reale, M., Horowitz, A., Liu, P., & Girard, J. M. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.06.002 · Cited by 745
Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning
Ng, H.-W., Nguyen, V. D., Vonikakis, V., & Winkler, S. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830593 · Cited by 657 · cited 1×
“Therefore, this sub-section briefly introduces some popular databases related to FER consisting of 2D and 3D video sequences and still images:”
BackgroundBenitez-Quiroz, C. F., Srinivasan, R., & Martinez, A. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.600 · Cited by 607
Mechrabian, A. (1968) · Medical Entomology and Zoology · Cited by 459
Pedestrian detection aided by deep learning semantic tasks
Tian, Y., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299143 · Cited by 407
Recurrent Neural Networks for Emotion Recognition in Video
Ebrahimi Kahou, S., Michalski, V., Konda, K., Memisevic, R., & Pal, C. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830596 · Cited by 379
Static and dynamic 3D facial expression recognition: A comprehensive survey
Sandbach, G., Zafeiriou, S., Pantic, M., & Yin, L. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.005 · Cited by 357
Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis
Liu, M., Li, S., Shan, S., Wang, R., & Chen, X. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16817-3_10 · Cited by 329
Deep Region and Multi-label Learning for Facial Action Unit Detection
Zhao, K., Chu, W.-S., & Zhang, H. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.369 · Cited by 329
Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor
Polikovsky, S., Kameda, Y., & Ohta, Y. (2009) · 3rd International Conference on Imaging for Crime Detection and Prevention (ICDP 2009) · doi:10.1049/ic.2009.0244 · Cited by 312
Ghimire, D., & Lee, J. (2013) · Sensors · doi:10.3390/s130607714 · Cited by 291
Hamm, J., Kohler, C. G., Gur, R. C., & Verma, R. (2011) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2011.06.023 · Cited by 288
Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks
Hasani, B., & Mahoor, M. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.282 · Cited by 287
Kim, D. H., Baddar, W. J., Jang, J., & Ro, Y. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2695999 · Cited by 257 · cited 3×
“Therefore, this sub-section briefly introduces some popular databases related to FER consisting of 2D and 3D video sequences and still images:”
BackgroundAction unit detection using sparse appearance descriptors in space-time video volumes
Jiang, B., Valstar, M. F., & Pantic, M. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771416 · Cited by 233
Chen, C.-H., Lee, I.-J., & Lin, L.-Y. (2015) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2014.10.015 · Cited by 229
AU-inspired Deep Networks for Facial Expression Feature Learning
Liu, M., Li, S., Shan, S., & Chen, X. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.02.011 · Cited by 224
AU-aware Deep Networks for facial expression recognition
Liu, M., Li, S., Shan, S., & Chen, X. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553734 · Cited by 218
Joint patch and multi-label learning for facial action unit detection
Zhao, K., Chu, W.-S., De la Torre, F., Cohn, J. F., & Zhang, H. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298833 · Cited by 196
Kaulard, K., Cunningham, D. W., Bülthoff, H. H., & Wallraven, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0032321 · Cited by 184
Siddiqi, M. H., Ali, R., Khan, A. M., Park, Y.-T., & Lee, S. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2405346 · Cited by 183
Deep Structured Learning for Facial Action Unit Intensity Estimation
Walecki, R., Rudovic, O. O., Pavlović, V., Schuller, B. W., & Pantić, M. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.605 · Cited by 174
A real time facial expression classification system using Local Binary Patterns
Happy, S. L., George, A., & Routray, A. (2012) · Intelligent Human Computer Interaction · doi:10.1109/ihci.2012.6481802 · Cited by 166
la Torre, F. D., Chu, W.-S., Xiong, X., Vicente, F., Ding, X., & Cohn, J. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163082 · Cited by 160
Facial expression recognition based on local region specific features and support vector machines
Ghimire, D., Jeong, S., Lee, J., & Park, S. H. (2016) · Multimedia Tools and Applications · doi:10.1007/s11042-016-3418-y · Cited by 154
Driver drowsiness detection using face expression recognition
Assari, M. A., & Rahmati, M. (2011) · IEEE International Conference on Signal and Image Processing Applications (ICSIPA) · doi:10.1109/icsipa.2011.6144162 · Cited by 150
Framework for reliable, real-time facial expression recognition for low resolution images
Khan, R. A., Meyer, A., Konik, H., & Bouakaz, S. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2013.03.022 · Cited by 148
Learning Spatial and Temporal Cues for Multi-Label Facial Action Unit Detection
Chu, W.-S., de la Torre, F. N., & Cohn, J. F. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.13 · Cited by 145
Real-Time Mobile Facial Expression Recognition System -- A Case Study
Suk, M., & Prabhakaran, B. (2014) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2014.25 · Cited by 136
A review of emotion recognition methods based on keystroke dynamics and mouse movements
Kolakowska, A. (2013) · 2013 6th International Conference on Human System Interactions (HSI) · doi:10.1109/hsi.2013.6577879 · Cited by 130
Multi angle optimal pattern-based deep learning for automatic facial expression recognition
Jain, D. K., Zhang, Z., & Huang, K. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.06.025 · Cited by 103
Shape analysis of local facial patches for 3D facial expression recognition
Maalej, A., Amor, B. B., Daoudi, M., Srivastava, A., & Berretti, S. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2011.02.012 · Cited by 97
Facial Expression Recognition Based on Local Phase Quantization and Sparse Representation
Wang, Z., & Ying, Z. (2012) · International Conference on Natural Computation (ICNC) · doi:10.1109/icnc.2012.6234551 · Cited by 78
Robust Facial Expression Recognition via Compressive Sensing
Zhang, S., Zhao, X., & Lei, B. (2012) · Sensors · doi:10.3390/s120303747 · Cited by 75
A new method for facial expression recognition based on sparse representation plus LBP
Huang, M., Wang, Z., & Ying, Z. (2010) · International Congress on Image and Signal Processing (CISP) · doi:10.1109/cisp.2010.5647898 · Cited by 74
A Real‐Time Facial Expression Recognition System for Online Games
Zhan, C., Li, W., Ogunbona, P., & Safaei, F. (2008) · International Journal of Computer Games Technology · doi:10.1155/2008/542918 · Cited by 73
Facial Action Unit Event Detection by Cascade of Tasks
Ding, X., Chu, W.-S., de la Torre, F. N., Cohn, J. F., & Wang, Q. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.298 · Cited by 73
Survey on real‐time facial expression recognition techniques
Deshmukh, S., Patwardhan, M. S., & Mahajan, A. (2016) · IET Biometrics · doi:10.1049/iet-bmt.2014.0104 · Cited by 62
Face Expression Detection on Kinect Using Active Appearance Model and Fuzzy Logic
Sujono, & Gunawan, A. A. S. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.07.558 · Cited by 55
Lee, S. H., Baddar, W. J., & Ro, Y. M. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2015.12.016 · Cited by 55
Facial emotion recognition using depth data
Szwoch, M., & Pieniążek, P. (2015) · Conference on Human System Interactions · doi:10.1109/hsi.2015.7170679 · Cited by 47
Facial Expression Recognition with Recurrent Neural Networks
Graves, A., Mayer, C., Wimmer, M., Schmidhuber, J., & Radig, B. (2008) · Cited by 36
Competitive affective gaming: winning with a smile
Mourão, A., & Magalhães, J. (2013) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2502081.2502115 · Cited by 34
Efficient Facial Expression Recognition for Human Robot Interaction
Dornaika, F., & Raducanu, B. (2007) · Springer eBooks · doi:10.1007/978-3-540-73007-1_84 · Cited by 29
Facial Expression Recognition from Infrared Thermal Videos
Shen, P., Wang, S., & Liu, Z. (2012) · Advances in intelligent systems and computing · doi:10.1007/978-3-642-33932-5_31 · Cited by 26
A Quick Review of Deep Learning in Facial Expression
Ghayoumi, M. (2017) · Journal of Communication and Computer · doi:10.17265/1548-7709/2017.01.004 · Cited by 26
Cited by
Artificial intelligence in the creative industries: a review
Anantrasirichai, N., & Bull, D. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10039-7 · Cited by 793
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 615
Liu, W., Qiu, J.-L., Zheng, W.-L., & Lu, B.-L. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3071170 · Cited by 452
Extended deep neural network for facial emotion recognition
Jain, D. K., Shamsolmoali, P., & Sehdev, P. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.01.008 · Cited by 400
Gu, X., Cao, Z., Jolfaei, A., Xu, P., Wu, D., Jung, T.-P., & Lin, C.-T. (2021) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2021.3052811 · Cited by 395
Metadata
- Authors
- Ko, B.
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2018-01-30
- Volume
- 18
- Issue
- 2
- Pages
- 401
- ISSN
- 1424-8220 (ISSN-L)
- OpenAlex
- W2792191740
- Cited by
- 724
- References
- 60
- FWCI
- 80.52
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.