Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 160

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Facial Emotion Recognition Using Conventional Machine Learning and Deep Learning Methods: Current Achievements, Analysis and Remaining Challenges

Open access
Khan, Amjad Rehman · 2022 · Information

Abstract

Facial emotion recognition (FER) is an emerging and significant research area in the pattern recognition domain. In daily life, the role of non-verbal communication is significant, and in overall communication, its involvement is around 55% to 93%. Facial emotion analysis is efficiently used in surveillance videos, expression analysis, gesture recognition, smart homes, computer games, depression treatment, patient monitoring, anxiety, detecting lies, psychoanalysis, paralinguistic communication, detecting operator fatigue and robotics. In this paper, we present a detailed review on FER. The literature is collected from different reputable research published during the current decade. This review is based on conventional machine learning (ML) and various deep learning (DL) approaches. Further, different FER datasets for evaluation metrics that are publicly available are discussed and compared with benchmark results. This paper provides a holistic review of FER using traditional ML and DL methods to highlight the future gap in this domain for new researchers. Finally, this review work is a guidebook and very helpful for young researchers in the FER area, providing a general understating and basic knowledge of the current state-of-the-art methods, and to experienced researchers looking for productive directions for future work.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/info13060268

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 91

  1. Jabeen, S., Mehmood, Z., Mahmood, T., Saba, T., Rehman, A., and Mahmood, M.T. (2018). An effective content-based image retrieval technique for image visuals representation based on the bag-of-visual-words model. PLoS ONE, 13. · DOI
  2. Moret-Tatay, C., Wester, A.G., and Gamermann, D. (2020). To Google or not: Differences on how online searches predict names and faces. Mathematics, 8. · DOI
  3. Ubaid, M.T., Khalil, M., Khan, M.U.G., Saba, T., and Rehman, A. (2021, January 13–14). Beard and Hair Detection, Segmentation and Changing Color Using Mask R-CNN. Proceedings of the International Conference on Information Technology and Applications, Dubai, United Arab Emirates. · doi:10.1007/978-981-16-7618-5_6
  4. Meethongjan, 2013, An intelligent fused approach for face recognition, J. Intell. Syst., 22, 197
  5. Rehman, 2011, Video motion perception using optimized Gabor filter, Int. J. Phys. Sci., 6, 2799
  6. Joudaki, 2022, Dynamic hand gesture recognition of sign language using geometric features learning, Int. J. Comput. Vis. Robot., 12, 1, 10.1504/IJCVR.2022.119239 · doi:10.1504/ijcvr.2022.119239
  7. Abunadi, I., Albraikan, A.A., Alzahrani, J.S., Eltahir, M.M., Hilal, A.M., Eldesouki, M.I., Motwakel, A., and Yaseen, I. (2022). An Automated Glowworm Swarm Optimization with an Inception-Based Deep Convolutional Neural Network for COVID-19 Diagnosis and Classification. Healthcare, 10. · DOI
  8. Yar, 2021, Vision sensor-based real-time fire detection in resource-constrained IoT environments, Comput. Intell. Neurosci., 2021, 5195508, 10.1155/2021/5195508 · DOI
  9. Yasin, 2010, Analysis of Internet Download Manager for collection of digital forensic artefacts, Digit. Investig., 7, 90, 10.1016/j.diin.2010.08.005 · doi:10.1016/j.diin.2010.08.005
  10. Rehman, 2014, Virtual machine security challenges: Case studies, Int. J. Mach. Learn. Cybern., 5, 729, 10.1007/s13042-013-0166-4 · DOI
  11. Afza, 2021, A framework of human action recognition using length control features fusion and weighted entropy-variances based feature selection, Image Vis. Comput., 106, 104090, 10.1016/j.imavis.2020.104090 · DOI
  12. Rehman, 2021, Microscopic brain tumor detection and classification using 3D CNN and feature selection architecture, Microsc. Res. Tech., 84, 133, 10.1002/jemt.23597 · DOI · Paper page
  13. Haji, 2019, Content-based image retrieval: A deep look at features prospectus, Int. J. Comput. Vis. Robot., 9, 14, 10.1504/IJCVR.2019.098004 · doi:10.1504/ijcvr.2019.098004
  14. Alkawaz, 2014, Facial animations: Future research directions & challenges, 3D Res., 5, 12, 10.1007/s13319-014-0012-9 · DOI
  15. Saleem, 2022, Efficient facial recognition authentication using edge and density variant sketch generator, CMC-Comput. Mater. Contin., 70, 505
  16. Rahim, 2014, Extreme facial expressions classification based on reality parameters, 3D Res., 5, 22, 10.1007/s13319-014-0022-7 · doi:10.1007/s13319-014-0022-7
  17. Rashid, M., Khan, M.A., Alhaisoni, M., Wang, S.H., Naqvi, S.R., Rehman, A., and Saba, T. (2020). A sustainable deep learning framework for object recognition using multi-layers deep features fusion and selection. Sustainability, 12. · DOI
  18. Lung, 2014, Fuzzy phoneme classification using multi-speaker vocal tract length normalization, IETE Tech. Rev., 31, 128, 10.1080/02564602.2014.892669 · DOI
  19. Amin, 2020, Brain tumor detection: A long short-term memory (LSTM)-based learning model, Neural Comput. Appl., 32, 15965, 10.1007/s00521-019-04650-7 · DOI · Paper page
  20. Ghayoumi, 2017, A quick review of deep learning in facial expression, J. Commun. Comput., 14, 34
  21. Ko, B.C. (2018). A brief review of facial emotion recognition based on visual information. Sensors, 18. · DOI · Paper page
  22. Sharif, 2017, Face recognition: A survey, J. Eng. Sci. Technol. Rev., 10, 166, 10.25103/jestr.102.20 · DOI
  23. Khan, 2020, A realistic image generation of face from text description using the fully trained generative adversarial networks, IEEE Access, 9, 1250, 10.1109/ACCESS.2020.3015656 · DOI
  24. Cornejo, J.Y.R., Pedrini, H., and Flórez-Revuelta, F. (2015). Facial expression recognition with occlusions based on geometric representation. Iberoamerican Congress on Pattern Recognition, Springer. · DOI
  25. Mahata, J., and Phadikar, A. (2017, January 23–24). Recent advances in human behaviour understanding: A survey. Proceedings of the Devices for Integrated Circuit (DevIC), Kalyani, India. · DOI
  26. Siddiqi, 2015, Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection, Multimed. Syst., 21, 541, 10.1007/s00530-014-0400-2 · DOI
  27. Varma, S., Shinde, M., and Chavan, S.S. (2020). Analysis of PCA and LDA features for facial expression recognition using SVM and HMM classifiers. Techno-Societal 2018, Springer. · DOI
  28. Reddy, 2019, Facial Emotion Recognition Using NLPCA and SVM, Traitement du Signal, 36, 13, 10.18280/ts.360102 · DOI
  29. Sajjad, 2019, Raspberry Pi assisted facial expression recognition framework for smart security in law-enforcement services, Inf. Sci., 479, 416, 10.1016/j.ins.2018.07.027 · DOI
  30. Nazir, 2018, Facial expression recognition using histogram of oriented gradients based transformed features, Clust. Comput., 21, 539, 10.1007/s10586-017-0921-5 · DOI
  31. Zeng, 2018, Facial expression recognition via learning deep sparse autoencoders, Neurocomputing, 273, 643, 10.1016/j.neucom.2017.08.043 · DOI · Paper page
  32. Uddin, 2017, A facial expression recognition system using robust face features from depth videos and deep learning, Comput. Electr. Eng., 63, 114, 10.1016/j.compeleceng.2017.04.019 · DOI
  33. Saleh, 2017, Geometric-based feature extraction and classification for emotion expressions of 3D video film, J. Adv. Inf. Technol., 8, 74
  34. Ghimire, 2017, Recognition of facial expressions based on salient geometric features and support vector machines, Multimed. Tools Appl., 76, 7921, 10.1007/s11042-016-3428-9 · DOI
  35. Wang, J., and Yang, H. (2008, January 27–30). Face detection based on template matching and 2DPCA algorithm. Proceedings of the 2008 Congress on Image and Signal Processing, Sanya, China. · doi:10.1109/cisp.2008.270
  36. Wu, 2017, Spontaneous versus posed smile recognition via region-specific texture descriptor and geometric facial dynamics, Front. Inf. Technol. Electron. Eng., 18, 955, 10.1631/FITEE.1600041 · DOI
  37. Acevedo, D., Negri, P., Buemi, M.E., Fernández, F.G., and Mejail, M. (June, January 30). A simple geometric-based descriptor for facial expression recognition. Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA. · DOI
  38. Kim, 2016, Facial expression recognition using ASM-based post-processing technique, Pattern Recognit. Image Anal., 26, 576, 10.1134/S105466181603010X · doi:10.1134/s105466181603010x
  39. Chang, K.Y., Chen, C.S., and Hung, Y.P. (2013, January 13–16). Intensity rank estimation of facial expressions based on a single image. Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC), Manchester, UK. · DOI
  40. Abbas, 2018, Machine aided malaria parasitemia detection in Giemsa-stained thin blood smears, Neural Comput. Appl., 29, 803, 10.1007/s00521-016-2474-6 · DOI
  41. Iqbal, 2019, Deep learning model integrating features and novel classifiers fusion for brain tumor segmentation, Microsc. Res. Tech., 82, 1302, 10.1002/jemt.23281 · DOI
  42. Khan, 2021, Prediction of COVID-19-pneumonia based on selected deep features and one class kernel extreme learning machine, Comput. Electr. Eng., 90, 106960, 10.1016/j.compeleceng.2020.106960 · DOI
  43. Khan, M.A., Sharif, M.I., Raza, M., Anjum, A., Saba, T., and Shad, S.A. (2019). Skin lesion segmentation and classification: A unified framework of deep neural network features fusion and selection. Expert Syst., e12497. · DOI · Paper page
  44. Nawaz, 2022, Skin cancer detection from dermoscopic images using deep learning and fuzzy k-means clustering, Microsc. Res. Tech., 85, 339, 10.1002/jemt.23908 · DOI · Paper page
  45. Rehman, 2018, Microscopic malaria parasitemia diagnosis and grading on benchmark datasets, Microsc. Res. Tech., 81, 1042, 10.1002/jemt.23071 · DOI · Paper page
  46. Saba, 2020, Secure and energy-efficient framework using Internet of Medical Things for e-healthcare, J. Infect. Public Health, 13, 1567, 10.1016/j.jiph.2020.06.027 · DOI
  47. Saba, 2018, Fundus image classification methods for the detection of glaucoma: A review, Microsc. Res. Tech., 81, 1105, 10.1002/jemt.23094 · DOI
  48. Saba, 2019, Automated lung nodule detection and classification based on multiple classifiers voting, Microsc. Res. Tech., 8, 1601, 10.1002/jemt.23326 · DOI
  49. Sadad, 2018, Fuzzy C-means and region growing based classification of tumor from mammograms using hybrid texture feature, J. Comput. Sci., 29, 34, 10.1016/j.jocs.2018.09.015 · DOI
  50. Amin, 2019, A new approach for brain tumor segmentation and classification based on score level fusion using transfer learning, J. Med. Syst., 43, 326, 10.1007/s10916-019-1453-8 · DOI · Paper page
  51. Javed, 2020, A comparative study of features selection for skin lesion detection from dermoscopic images, Netw. Modeling Anal. Health Inform. Bioinform., 9, 4, 10.1007/s13721-019-0209-1 · DOI
  52. Li, 2019, Micro-expression recognition based on 3D flow convolutional neural network, Pattern Anal. Appl., 22, 1331, 10.1007/s10044-018-0757-5 · DOI · Paper page
  53. Li, B.Y., Mian, A.S., Liu, W., and Krishna, A. (2013, January 15–17). Using kinect for face recognition under varying poses, expressions, illumination and disguise. Proceedings of the IEEE workshop on applications of computer vision (WACV), Clearwater Beach, FL, USA. · doi:10.1109/wacv.2013.6475017
  54. Breuer, R., and Kimmel, R. (2017). A deep learning perspective on the origin of facial expressions. arXiv.
  55. Chu, W.S., De La Torre, F., and Cohn, J.F. (June, January 30). Learning spatial and temporal cues for multi-label facial action unit detection. Proceedings of the 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA. · DOI
  56. Hasani, B., and Mahoor, M.H. (2017, January 21–26). Facial expression recognition using enhanced deep 3D convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA. · DOI · Paper page
  57. Zhang, 2017, Facial expression recognition based on deep evolutional spatial-temporal networks, IEEE Trans. Image Processing, 26, 4193, 10.1109/TIP.2017.2689999 · DOI · Paper page
  58. Jain, 2020, Multi angle optimal pattern-based deep learning for automatic facial expression recognition, Pattern Recognit. Lett., 139, 157, 10.1016/j.patrec.2017.06.025 · DOI
  59. Al-Shabi, M., Cheah, W.P., and Connie, T. (2016). Facial Expression Recognition Using a Hybrid CNN-SIFT Aggregator. arXiv.
  60. Yu, Z., and Zhang, C. (2015, January 9–13). Image based static facial expression recognition with multiple deep network learning. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA. · DOI · Paper page
  61. Li, 2020, Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy, Vis. Comput., 36, 391, 10.1007/s00371-019-01627-4 · DOI
  62. Xie, 2019, Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition, Pattern Recognit., 92, 177, 10.1016/j.patcog.2019.03.019 · DOI · Paper page
  63. Nasir, I.M., Raza, M., Shah, J.H., Khan, M.A., and Rehman, A. (2021, January 6–7). Human action recognition using machine learning in uncontrolled environment. Proceedings of the 2021 1st International Conference on Artificial Intelligence and Data Analytics (CAIDA), Riyadh, Saudi Arabia. · DOI
  64. Rehman, A., Haseeb, K., Saba, T., Lloret, J., and Tariq, U. (2021). Secured Big Data Analytics for Decision-Oriented Medical System Using Internet of Things. Electronics, 10. · DOI
  65. Harouni, 2014, Online Persian/Arabic script classification without contextual information, Imaging Sci. J., 62, 437, 10.1179/1743131X14Y.0000000083 · doi:10.1179/1743131x14y.0000000083
  66. Iftikhar, 2017, An evolution based hybrid approach for heart diseases classification and associated risk factors identification, Biomed. Res., 28, 3451
  67. Jamal, 2017, Retinal imaging analysis based on vessel detection, Microsc. Res. Tech., 80, 799, 10.1002/jemt.22867 · DOI
  68. Neamah, 2014, Discriminative features mining for offline handwritten signature verification, 3D Res., 5, 2, 10.1007/s13319-013-0002-3 · doi:10.1007/s13319-013-0002-3
  69. Ramzan, 2020, Volumetric segmentation of brain regions from MRI scans using 3D convolutional neural networks, IEEE Access, 8, 103697, 10.1109/ACCESS.2020.2998901 · DOI
  70. Keltner, 2019, Emotional expression: Advances in basic emotion theory, J. Nonverbal Behav., 43, 133, 10.1007/s10919-019-00293-3 · DOI · Paper page
  71. Phetchanchai, 2010, Index financial time series based on zigzag-perceptually important points, J. Comput. Sci., 6, 1389, 10.3844/jcssp.2010.1389.1395 · DOI
  72. Saba, 2014, Annotated comparisons of proposed preprocessing techniques for script recognition, Neural Comput. Appl., 25, 1337, 10.1007/s00521-014-1618-9 · DOI
  73. Saba, 2013, Analysis of vision based systems to detect real time goal events in soccer videos, Appl. Artif. Intell., 27, 656, 10.1080/08839514.2013.787779 · doi:10.1080/08839514.2013.787779
  74. Ullah, 2019, An ensemble classification of exudates in color fundus images using an evolutionary algorithm based optimal features selection, Microsc. Res. Tech., 82, 361, 10.1002/jemt.23178 · DOI
  75. Sharif, 2019, Scene analysis and search using local features and support vector machine for effective content-based image retrieval, Artif. Intell. Rev., 52, 901, 10.1007/s10462-018-9636-0 · DOI
  76. Yousaf, 2019, Mobile-health applications for the efficient delivery of health care facility to people with dementia (PwD) and support to their carers: A survey, BioMed Res. Int., 2019, 7151475, 10.1155/2019/7151475 · DOI
  77. Saba, 2014, Evaluation of current documents image denoising techniques: A comparative study, Appl. Artif. Intell., 28, 879, 10.1080/08839514.2014.954344 · DOI
  78. Saba, 2011, Improved statistical features for cursive character recognition, Int. J. Innov. Comput. Inf. Control., 7, 5211
  79. Rehman, 2011, Document skew estimation and correction: Analysis of techniques, common problems and possible solutions, Appl. Artif. Intell., 25, 769, 10.1080/08839514.2011.607009 · doi:10.1080/08839514.2011.607009
  80. Rehman, 2011, An automatic approach for line detection and removal without smash-up characters, Imaging Sci. J., 59, 177, 10.1179/136821910X12863758415649 · doi:10.1179/136821910x12863758415649
  81. Saba, 2022, IoMT Enabled Melanoma Detection Using Improved Region Growing Lesion Boundary Extraction, CMC-Comput. Mater. Contin., 71, 6219
  82. Yousaf, 2020, A comprehensive study of mobile-health based assistive technology for the healthcare of dementia and Alzheimer’s disease (AD), Health Care Manag. Sci., 23, 287, 10.1007/s10729-019-09486-0 · DOI
  83. Ahmad, 2014, Data hiding based on improved exploiting modification direction method and Huffman coding, J. Intell. Syst., 23, 451
  84. Rehman, 2017, Ear biometrics for human classification based on region features mining, Biomed. Res., 28, 4660
  85. Rahim, 2017, 3D bones segmentation based on CT images visualization, Biomed. Res., 28, 3641
  86. Nodehi, 2014, Intelligent fuzzy approach for fast fractal image compression, EURASIP J. Adv. Signal Processing, 2014, 112, 10.1186/1687-6180-2014-112 · doi:10.1186/1687-6180-2014-112
  87. Haron, 2012, Parameterization method on B-spline curve, Math. Probl. Eng., 2012, 640472, 10.1155/2012/640472 · doi:10.1155/2012/640472
  88. Rehman, 2011, Performance analysis of character segmentation approach for cursive script recognition on benchmark database, Digit. Signal Processing, 21, 486, 10.1016/j.dsp.2011.01.016 · doi:10.1016/j.dsp.2011.01.016
  89. Yousuf, 2018, A novel technique based on visual words fusion analysis of sparse features for effective content-based image retrieval, Math. Probl. Eng., 2018, 13, 10.1155/2018/2134395 · DOI
  90. Saba, 2019, Cloud-based decision support system for the detection and classification of malignant cells in breast cancer using breast cytology images, Microsc. Res. Tech., 82, 775, 10.1002/jemt.23222 · DOI
  91. Yang, 2017, Facial expression recognition based on dual-feature fusion and improved random forest classifier, Multimed. Tools Appl., 9, 20477

Metadata

Authors
Khan, A. R.
Venue
Information
Publisher
MDPI AG
Published
2022-05-25
Volume
13
Issue
6
Pages
268
ISSN
2078-2489 (ISSN-L)
OpenAlex
W4281488040
Cited by
160
References
5
FWCI
29.98
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.