The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Facial Emotion Recognition Using Conventional Machine Learning and Deep Learning Methods: Current Achievements, Analysis and Remaining Challenges
Open accessAbstract
Facial emotion recognition (FER) is an emerging and significant research area in the pattern recognition domain. In daily life, the role of non-verbal communication is significant, and in overall communication, its involvement is around 55% to 93%. Facial emotion analysis is efficiently used in surveillance videos, expression analysis, gesture recognition, smart homes, computer games, depression treatment, patient monitoring, anxiety, detecting lies, psychoanalysis, paralinguistic communication, detecting operator fatigue and robotics. In this paper, we present a detailed review on FER. The literature is collected from different reputable research published during the current decade. This review is based on conventional machine learning (ML) and various deep learning (DL) approaches. Further, different FER datasets for evaluation metrics that are publicly available are discussed and compared with benchmark results. This paper provides a holistic review of FER using traditional ML and DL methods to highlight the future gap in this domain for new researchers. Finally, this review work is a guidebook and very helpful for young researchers in the FER area, providing a general understating and basic knowledge of the current state-of-the-art methods, and to experienced researchers looking for productive directions for future work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303
Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism
Li, Y., Zeng, J., Shan, S., & Chen, X. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2886767 · Cited by 916 · cited 2×
“) was suggested by Li et al. [55] that could interpret the occlusion regions of the face and concentrate on Yu and Zhang [64] used a hybrid CNN to execute FER and, in 2015, obtained state-of-the-art outcomes in FER. more discriminatory, non-occluded regions.”
MethodLopes, A. T., de Aguiar, E., De Souza, A. F., & Oliveira-Santos, T. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.07.026 · Cited by 799 · cited 2×
“Lopes et al. [56] CK+ JAFFE CNN and specific image This dataset consists of 2D and 3D faces for emotion recognition, facial action detection, 3D face reconstruction, etc. BU-3DFE This dataset is different from other datasets in the following aspects. preprocessing steps Seven emotions There are 105…”
BackgroundMethodJoint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition
Jung, H., Lee, S., Yim, J., Park, S., & Kim, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.341 · Cited by 781
A review of emotion recognition methods based on keystroke dynamics and mouse movements
Kolakowska, A. (2013) · 2013 6th International Conference on Human System Interactions (HSI) · doi:10.1109/hsi.2013.6577879 · Cited by 130
Research works printed but not linked: 91
- Jabeen, S., Mehmood, Z., Mahmood, T., Saba, T., Rehman, A., and Mahmood, M.T. (2018). An effective content-based image retrieval technique for image visuals representation based on the bag-of-visual-words model. PLoS ONE, 13. · DOI
- Moret-Tatay, C., Wester, A.G., and Gamermann, D. (2020). To Google or not: Differences on how online searches predict names and faces. Mathematics, 8. · DOI
- Ubaid, M.T., Khalil, M., Khan, M.U.G., Saba, T., and Rehman, A. (2021, January 13–14). Beard and Hair Detection, Segmentation and Changing Color Using Mask R-CNN. Proceedings of the International Conference on Information Technology and Applications, Dubai, United Arab Emirates. · doi:10.1007/978-981-16-7618-5_6
- Meethongjan, 2013, An intelligent fused approach for face recognition, J. Intell. Syst., 22, 197
- Rehman, 2011, Video motion perception using optimized Gabor filter, Int. J. Phys. Sci., 6, 2799
- Joudaki, 2022, Dynamic hand gesture recognition of sign language using geometric features learning, Int. J. Comput. Vis. Robot., 12, 1, 10.1504/IJCVR.2022.119239 · doi:10.1504/ijcvr.2022.119239
- Abunadi, I., Albraikan, A.A., Alzahrani, J.S., Eltahir, M.M., Hilal, A.M., Eldesouki, M.I., Motwakel, A., and Yaseen, I. (2022). An Automated Glowworm Swarm Optimization with an Inception-Based Deep Convolutional Neural Network for COVID-19 Diagnosis and Classification. Healthcare, 10. · DOI
- Yar, 2021, Vision sensor-based real-time fire detection in resource-constrained IoT environments, Comput. Intell. Neurosci., 2021, 5195508, 10.1155/2021/5195508 · DOI
- Yasin, 2010, Analysis of Internet Download Manager for collection of digital forensic artefacts, Digit. Investig., 7, 90, 10.1016/j.diin.2010.08.005 · doi:10.1016/j.diin.2010.08.005
- Rehman, 2014, Virtual machine security challenges: Case studies, Int. J. Mach. Learn. Cybern., 5, 729, 10.1007/s13042-013-0166-4 · DOI
- Afza, 2021, A framework of human action recognition using length control features fusion and weighted entropy-variances based feature selection, Image Vis. Comput., 106, 104090, 10.1016/j.imavis.2020.104090 · DOI
- Rehman, 2021, Microscopic brain tumor detection and classification using 3D CNN and feature selection architecture, Microsc. Res. Tech., 84, 133, 10.1002/jemt.23597 · DOI · Paper page
- Haji, 2019, Content-based image retrieval: A deep look at features prospectus, Int. J. Comput. Vis. Robot., 9, 14, 10.1504/IJCVR.2019.098004 · doi:10.1504/ijcvr.2019.098004
- Alkawaz, 2014, Facial animations: Future research directions & challenges, 3D Res., 5, 12, 10.1007/s13319-014-0012-9 · DOI
- Saleem, 2022, Efficient facial recognition authentication using edge and density variant sketch generator, CMC-Comput. Mater. Contin., 70, 505
- Rahim, 2014, Extreme facial expressions classification based on reality parameters, 3D Res., 5, 22, 10.1007/s13319-014-0022-7 · doi:10.1007/s13319-014-0022-7
- Rashid, M., Khan, M.A., Alhaisoni, M., Wang, S.H., Naqvi, S.R., Rehman, A., and Saba, T. (2020). A sustainable deep learning framework for object recognition using multi-layers deep features fusion and selection. Sustainability, 12. · DOI
- Lung, 2014, Fuzzy phoneme classification using multi-speaker vocal tract length normalization, IETE Tech. Rev., 31, 128, 10.1080/02564602.2014.892669 · DOI
- Amin, 2020, Brain tumor detection: A long short-term memory (LSTM)-based learning model, Neural Comput. Appl., 32, 15965, 10.1007/s00521-019-04650-7 · DOI · Paper page
- Ghayoumi, 2017, A quick review of deep learning in facial expression, J. Commun. Comput., 14, 34
- Ko, B.C. (2018). A brief review of facial emotion recognition based on visual information. Sensors, 18. · DOI · Paper page
- Sharif, 2017, Face recognition: A survey, J. Eng. Sci. Technol. Rev., 10, 166, 10.25103/jestr.102.20 · DOI
- Khan, 2020, A realistic image generation of face from text description using the fully trained generative adversarial networks, IEEE Access, 9, 1250, 10.1109/ACCESS.2020.3015656 · DOI
- Cornejo, J.Y.R., Pedrini, H., and Flórez-Revuelta, F. (2015). Facial expression recognition with occlusions based on geometric representation. Iberoamerican Congress on Pattern Recognition, Springer. · DOI
- Mahata, J., and Phadikar, A. (2017, January 23–24). Recent advances in human behaviour understanding: A survey. Proceedings of the Devices for Integrated Circuit (DevIC), Kalyani, India. · DOI
- Siddiqi, 2015, Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection, Multimed. Syst., 21, 541, 10.1007/s00530-014-0400-2 · DOI
- Varma, S., Shinde, M., and Chavan, S.S. (2020). Analysis of PCA and LDA features for facial expression recognition using SVM and HMM classifiers. Techno-Societal 2018, Springer. · DOI
- Reddy, 2019, Facial Emotion Recognition Using NLPCA and SVM, Traitement du Signal, 36, 13, 10.18280/ts.360102 · DOI
- Sajjad, 2019, Raspberry Pi assisted facial expression recognition framework for smart security in law-enforcement services, Inf. Sci., 479, 416, 10.1016/j.ins.2018.07.027 · DOI
- Nazir, 2018, Facial expression recognition using histogram of oriented gradients based transformed features, Clust. Comput., 21, 539, 10.1007/s10586-017-0921-5 · DOI
- Zeng, 2018, Facial expression recognition via learning deep sparse autoencoders, Neurocomputing, 273, 643, 10.1016/j.neucom.2017.08.043 · DOI · Paper page
- Uddin, 2017, A facial expression recognition system using robust face features from depth videos and deep learning, Comput. Electr. Eng., 63, 114, 10.1016/j.compeleceng.2017.04.019 · DOI
- Saleh, 2017, Geometric-based feature extraction and classification for emotion expressions of 3D video film, J. Adv. Inf. Technol., 8, 74
- Ghimire, 2017, Recognition of facial expressions based on salient geometric features and support vector machines, Multimed. Tools Appl., 76, 7921, 10.1007/s11042-016-3428-9 · DOI
- Wang, J., and Yang, H. (2008, January 27–30). Face detection based on template matching and 2DPCA algorithm. Proceedings of the 2008 Congress on Image and Signal Processing, Sanya, China. · doi:10.1109/cisp.2008.270
- Wu, 2017, Spontaneous versus posed smile recognition via region-specific texture descriptor and geometric facial dynamics, Front. Inf. Technol. Electron. Eng., 18, 955, 10.1631/FITEE.1600041 · DOI
- Acevedo, D., Negri, P., Buemi, M.E., Fernández, F.G., and Mejail, M. (June, January 30). A simple geometric-based descriptor for facial expression recognition. Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA. · DOI
- Kim, 2016, Facial expression recognition using ASM-based post-processing technique, Pattern Recognit. Image Anal., 26, 576, 10.1134/S105466181603010X · doi:10.1134/s105466181603010x
- Chang, K.Y., Chen, C.S., and Hung, Y.P. (2013, January 13–16). Intensity rank estimation of facial expressions based on a single image. Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC), Manchester, UK. · DOI
- Abbas, 2018, Machine aided malaria parasitemia detection in Giemsa-stained thin blood smears, Neural Comput. Appl., 29, 803, 10.1007/s00521-016-2474-6 · DOI
- Iqbal, 2019, Deep learning model integrating features and novel classifiers fusion for brain tumor segmentation, Microsc. Res. Tech., 82, 1302, 10.1002/jemt.23281 · DOI
- Khan, 2021, Prediction of COVID-19-pneumonia based on selected deep features and one class kernel extreme learning machine, Comput. Electr. Eng., 90, 106960, 10.1016/j.compeleceng.2020.106960 · DOI
- Khan, M.A., Sharif, M.I., Raza, M., Anjum, A., Saba, T., and Shad, S.A. (2019). Skin lesion segmentation and classification: A unified framework of deep neural network features fusion and selection. Expert Syst., e12497. · DOI · Paper page
- Nawaz, 2022, Skin cancer detection from dermoscopic images using deep learning and fuzzy k-means clustering, Microsc. Res. Tech., 85, 339, 10.1002/jemt.23908 · DOI · Paper page
- Rehman, 2018, Microscopic malaria parasitemia diagnosis and grading on benchmark datasets, Microsc. Res. Tech., 81, 1042, 10.1002/jemt.23071 · DOI · Paper page
- Saba, 2020, Secure and energy-efficient framework using Internet of Medical Things for e-healthcare, J. Infect. Public Health, 13, 1567, 10.1016/j.jiph.2020.06.027 · DOI
- Saba, 2018, Fundus image classification methods for the detection of glaucoma: A review, Microsc. Res. Tech., 81, 1105, 10.1002/jemt.23094 · DOI
- Saba, 2019, Automated lung nodule detection and classification based on multiple classifiers voting, Microsc. Res. Tech., 8, 1601, 10.1002/jemt.23326 · DOI
- Sadad, 2018, Fuzzy C-means and region growing based classification of tumor from mammograms using hybrid texture feature, J. Comput. Sci., 29, 34, 10.1016/j.jocs.2018.09.015 · DOI
- Amin, 2019, A new approach for brain tumor segmentation and classification based on score level fusion using transfer learning, J. Med. Syst., 43, 326, 10.1007/s10916-019-1453-8 · DOI · Paper page
- Javed, 2020, A comparative study of features selection for skin lesion detection from dermoscopic images, Netw. Modeling Anal. Health Inform. Bioinform., 9, 4, 10.1007/s13721-019-0209-1 · DOI
- Li, 2019, Micro-expression recognition based on 3D flow convolutional neural network, Pattern Anal. Appl., 22, 1331, 10.1007/s10044-018-0757-5 · DOI · Paper page
- Li, B.Y., Mian, A.S., Liu, W., and Krishna, A. (2013, January 15–17). Using kinect for face recognition under varying poses, expressions, illumination and disguise. Proceedings of the IEEE workshop on applications of computer vision (WACV), Clearwater Beach, FL, USA. · doi:10.1109/wacv.2013.6475017
- Breuer, R., and Kimmel, R. (2017). A deep learning perspective on the origin of facial expressions. arXiv.
- Chu, W.S., De La Torre, F., and Cohn, J.F. (June, January 30). Learning spatial and temporal cues for multi-label facial action unit detection. Proceedings of the 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA. · DOI
- Hasani, B., and Mahoor, M.H. (2017, January 21–26). Facial expression recognition using enhanced deep 3D convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA. · DOI · Paper page
- Zhang, 2017, Facial expression recognition based on deep evolutional spatial-temporal networks, IEEE Trans. Image Processing, 26, 4193, 10.1109/TIP.2017.2689999 · DOI · Paper page
- Jain, 2020, Multi angle optimal pattern-based deep learning for automatic facial expression recognition, Pattern Recognit. Lett., 139, 157, 10.1016/j.patrec.2017.06.025 · DOI
- Al-Shabi, M., Cheah, W.P., and Connie, T. (2016). Facial Expression Recognition Using a Hybrid CNN-SIFT Aggregator. arXiv.
- Yu, Z., and Zhang, C. (2015, January 9–13). Image based static facial expression recognition with multiple deep network learning. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA. · DOI · Paper page
- Li, 2020, Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy, Vis. Comput., 36, 391, 10.1007/s00371-019-01627-4 · DOI
- Xie, 2019, Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition, Pattern Recognit., 92, 177, 10.1016/j.patcog.2019.03.019 · DOI · Paper page
- Nasir, I.M., Raza, M., Shah, J.H., Khan, M.A., and Rehman, A. (2021, January 6–7). Human action recognition using machine learning in uncontrolled environment. Proceedings of the 2021 1st International Conference on Artificial Intelligence and Data Analytics (CAIDA), Riyadh, Saudi Arabia. · DOI
- Rehman, A., Haseeb, K., Saba, T., Lloret, J., and Tariq, U. (2021). Secured Big Data Analytics for Decision-Oriented Medical System Using Internet of Things. Electronics, 10. · DOI
- Harouni, 2014, Online Persian/Arabic script classification without contextual information, Imaging Sci. J., 62, 437, 10.1179/1743131X14Y.0000000083 · doi:10.1179/1743131x14y.0000000083
- Iftikhar, 2017, An evolution based hybrid approach for heart diseases classification and associated risk factors identification, Biomed. Res., 28, 3451
- Jamal, 2017, Retinal imaging analysis based on vessel detection, Microsc. Res. Tech., 80, 799, 10.1002/jemt.22867 · DOI
- Neamah, 2014, Discriminative features mining for offline handwritten signature verification, 3D Res., 5, 2, 10.1007/s13319-013-0002-3 · doi:10.1007/s13319-013-0002-3
- Ramzan, 2020, Volumetric segmentation of brain regions from MRI scans using 3D convolutional neural networks, IEEE Access, 8, 103697, 10.1109/ACCESS.2020.2998901 · DOI
- Keltner, 2019, Emotional expression: Advances in basic emotion theory, J. Nonverbal Behav., 43, 133, 10.1007/s10919-019-00293-3 · DOI · Paper page
- Phetchanchai, 2010, Index financial time series based on zigzag-perceptually important points, J. Comput. Sci., 6, 1389, 10.3844/jcssp.2010.1389.1395 · DOI
- Saba, 2014, Annotated comparisons of proposed preprocessing techniques for script recognition, Neural Comput. Appl., 25, 1337, 10.1007/s00521-014-1618-9 · DOI
- Saba, 2013, Analysis of vision based systems to detect real time goal events in soccer videos, Appl. Artif. Intell., 27, 656, 10.1080/08839514.2013.787779 · doi:10.1080/08839514.2013.787779
- Ullah, 2019, An ensemble classification of exudates in color fundus images using an evolutionary algorithm based optimal features selection, Microsc. Res. Tech., 82, 361, 10.1002/jemt.23178 · DOI
- Sharif, 2019, Scene analysis and search using local features and support vector machine for effective content-based image retrieval, Artif. Intell. Rev., 52, 901, 10.1007/s10462-018-9636-0 · DOI
- Yousaf, 2019, Mobile-health applications for the efficient delivery of health care facility to people with dementia (PwD) and support to their carers: A survey, BioMed Res. Int., 2019, 7151475, 10.1155/2019/7151475 · DOI
- Saba, 2014, Evaluation of current documents image denoising techniques: A comparative study, Appl. Artif. Intell., 28, 879, 10.1080/08839514.2014.954344 · DOI
- Saba, 2011, Improved statistical features for cursive character recognition, Int. J. Innov. Comput. Inf. Control., 7, 5211
- Rehman, 2011, Document skew estimation and correction: Analysis of techniques, common problems and possible solutions, Appl. Artif. Intell., 25, 769, 10.1080/08839514.2011.607009 · doi:10.1080/08839514.2011.607009
- Rehman, 2011, An automatic approach for line detection and removal without smash-up characters, Imaging Sci. J., 59, 177, 10.1179/136821910X12863758415649 · doi:10.1179/136821910x12863758415649
- Saba, 2022, IoMT Enabled Melanoma Detection Using Improved Region Growing Lesion Boundary Extraction, CMC-Comput. Mater. Contin., 71, 6219
- Yousaf, 2020, A comprehensive study of mobile-health based assistive technology for the healthcare of dementia and Alzheimer’s disease (AD), Health Care Manag. Sci., 23, 287, 10.1007/s10729-019-09486-0 · DOI
- Ahmad, 2014, Data hiding based on improved exploiting modification direction method and Huffman coding, J. Intell. Syst., 23, 451
- Rehman, 2017, Ear biometrics for human classification based on region features mining, Biomed. Res., 28, 4660
- Rahim, 2017, 3D bones segmentation based on CT images visualization, Biomed. Res., 28, 3641
- Nodehi, 2014, Intelligent fuzzy approach for fast fractal image compression, EURASIP J. Adv. Signal Processing, 2014, 112, 10.1186/1687-6180-2014-112 · doi:10.1186/1687-6180-2014-112
- Haron, 2012, Parameterization method on B-spline curve, Math. Probl. Eng., 2012, 640472, 10.1155/2012/640472 · doi:10.1155/2012/640472
- Rehman, 2011, Performance analysis of character segmentation approach for cursive script recognition on benchmark database, Digit. Signal Processing, 21, 486, 10.1016/j.dsp.2011.01.016 · doi:10.1016/j.dsp.2011.01.016
- Yousuf, 2018, A novel technique based on visual words fusion analysis of sparse features for effective content-based image retrieval, Math. Probl. Eng., 2018, 13, 10.1155/2018/2134395 · DOI
- Saba, 2019, Cloud-based decision support system for the detection and classification of malignant cells in breast cancer using breast cytology images, Microsc. Res. Tech., 82, 775, 10.1002/jemt.23222 · DOI
- Yang, 2017, Facial expression recognition based on dual-feature fusion and improved random forest classifier, Multimed. Tools Appl., 9, 20477
Cited by
Efficient Facial Expression Recognition Through Lightweight CNN Technique on Public Datasets
Grover, R., & Bansal, S. (2024) · SN Computer Science · doi:10.1007/s42979-024-03557-y · Cited by 6
Metadata
- Authors
- Khan, A. R.
- Venue
- Information
- Publisher
- MDPI AG
- Published
- 2022-05-25
- Volume
- 13
- Issue
- 6
- Pages
- 268
- ISSN
- 2078-2489 (ISSN-L)
- OpenAlex
- W4281488040
- Cited by
- 160
- References
- 5
- FWCI
- 29.98
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.