Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark

“[16] SFEW [15], JAFFE [30] and Multi-PIE [18] dataset SVM classification model using 3 datasets Images are similar to real world conditions The faces location is not available in the image which is useful for accurate initialization for feature extractors”

“…Wild Challenge and Workshop (EmotiW 2015) Challenge consists of two real-world condition emotion classification sub-challenges: 1) Audio-Video based emotion recognition challenge based on AFEW [8] database; 2) Image based static facial expression recognition challenge based on SFEW [9] database.”

“ion system. Although some video clips are clipped from movies, they are closer to the real world than the previous databases from lab conditions. In recent years, as more and more emotional databases [37,26,9,15] in the wild have been collected and annotated, research on automatic algorithms for emotion recognition in…”

“All of the facial photos in the database were edited from movies and included seven fundamental facial expressions SFEW [151] –”

“Due to its direct and intuitive definition of facial expressions, the categorical model is used in the majority of FER algorithms ([4], [1], [28], [26], etc) and large-scale databases (AffectNet [44], RAF-DB [36], SFEW [13], FER-2013 [22], EmotionNet [19], etc).”

“Although, on wild databases (RAF-DB [7] and SFEW [6]) the added value of both contributions is more noticeable, specially on SFEW [6] we can obtain a 16% gain on accuracy compared to the ViT without a SE neither a pre-training on FER-2013 [19].”

“0 [22], and test its generalization on three standard FER databases, i.e., Oulu-CASIA [23], CK+ [24], and MMI [25], to demonstrate the benefit from geometric knowledge to practical FER applications.”

“database called SFEW [7] was then created from AFEW [8] for facial expression analysis of an individual in images.”

“shown to have good performance in several recent works [15, 16, 32].”

“JAFFE [32] strong emotions 7 213 controlled SFEW [7] strong emotions 7 663 uncontrolled DISFA [34] strong emotions 7 4,845 controlled FER2013 [15] strong emotions 7 36K uncontrolled RAF-DB [23] strong emotions 18 30K uncontrolled EmotioNet [11] strong emotions 16 50K (950K)(1)uncontrolled AffectNet [37] strong emotion…”

“…[23], containing 12,271 training images annotated with emotion labels, with a large number of annotators for each sample and hence a high confidence in the annotations. ii) SFEW [10], containing fewer than 1,000 training images, but extracted from AFEW’s frames and annotated with the same labels.”

“0) [39] is a data corpus of dynamic temporal facial expressions labeled with neutral and the six typical expressions.”

“This performance saturation on standard benchmarks [7] has induced the collection of new and more challenging datasets.”

“The effect of progressive image resizing on the performance of the proposed system is reported in Table 5 and from this table, we can observe that for the KDEF, GENKI, CK+, and SFEW databases, the proposed FERS exhibits a better performance for F 192 × 192 × 3 images than for both F 96 × 96 × 3 and F 48 × 48 × 3 image…”

“To validate and keep the highest challenge to implement FER in real-environment scenarios, experiments have been carried out on three benchmark databases collected in the Wild, FER 2013 [24], SFEW 2.0 [25], and ExpW [26] datasets.”

“As the labeled facial images Table VI provides comparison results of our method with the state-of-the-art results reported in [7], [33], [79], [80] including GMLDA, MvDA, GMLPP, DS-PLVM, CG-FER, and the baseline designed by the dataset creators [19].”

“To validate the effectiveness of the proposed method, we evaluate the performance on three in-the-lab FER databases (CK+ (Lucey et al., 2010), MMI (Valstar and Pantic, 2010), and Oulu-CASIA (Zhao et al., 2011)) and four in-the-wild databases (RAF-DB (Li et al., 2017), SFEW (Dhall et al., 2011), Aff-Wild2 (Kollias and…”

“These certainties are used to train a RBF-kernel SVM to predict one of the eight basic emotional expressions (neutral, anger, contempt, disgust, fear, happiness, sadness, surprise) using four labeled data sets [15], [19], [20], [21] including both elicited and genuine expressions.”

“…public available databases in the FER research field: the Extended Cohn-Kanade (CK + ) database [45], the Japanese Female Facial Expressions (JAFFE) database [46], the Static Facial Expressions in the Wild (SFEW) database [47] and the Facial Expression Recognition 2013 (FER-2013) database [48].”

“We have found that the best results are achieved by first using the Model-4 [76] architecture trained on the RAFDB [77] dataset to detect disgust, followed by the Model-4 architecture trained on the SFEW [78] dataset to detect fear.”

“For the experimental purpose, KDEF (Karolinska-directed emotional faces) [26], Static Facial Expressions in the Wild 2.0 (SFEW 2.0) [27], and FER2013 [28] benchmark databases are employed to validate the stability and effectiveness of the proposed method.”

“SFEW [59] dataset is created by selecting static frames from the AFEW database.”

“In order to evaluate the performance of the RVLBP descriptor, we conduct experiments on the SFEW [5], GENKI-4K [31] and HAPPEI [7] databases, for the task of facial expression recognition in the wild, smile detection and happiness intensity estimation, respectively.”

“SFEW [46] is a static facial expression dataset selected from movies.”

“RAF-DB [18] and SFEW [39] are the most used databases representing the real-world facial expressions.”

“As a matter of fact, most current FER datasets are built upon these discrete categories, varying on the spe-cific definition of emotion concepts, such as JAFFE [21], CK+ [22], KDEF [23], SFEW [24], FER2013 [25], FER-Wild [26], AffectNet [16], and Aff-Wild2 [27].”

“The SFEW dataset [8] contains 879 training samples and 406 validation samples.”

“On AffectNet, RAF-DB, and SFEW datasets, our method achieves 62.03%, 88.23%, and 56.15% recognition accuracy.”

“At present, the facial expression databases commonly used at home and abroad are: JAFFE dataset [34], Fer2013 dataset [35], Cohn-Kanade dataset and CK + dataset [36], [37], RAF2017 dataset [38] And other data sets [39]–[41].”

“0 [5]) do not have AU annotations, and it is very expensive and labor-consuming to annotate the AUs for these datasets.”

“The five templates are constructed by averaging the shape of five manually grouped facial images from the SFEW dataset [36].”

“Several works have been proposed in the literature for facial emotions classification from static images [4, 5, 9, 20, 21, 32, 35, 47–49, 53, 59, 61, 65, 68, 74] or from continuous video input [1, 2, 43].”

“For instance, Extended Cohn-Kanade (CK +) [134], Static Facial Expressions in the Wild (SFEW) [135], and Toronto Face Database (TFD) [136] are a few publicly available databases that contain these basic expressions and serve as benchmarks for FER models.”

“(iii) Static face in the wild (SFEW): the SFEW [35] contains real-time movie images which are captured in unconstrained settings.”

“…several popular basic expression datasets, including three in-the-lab datasets (CK+ (Lucey et al. 2010), MMI (Pan-tic et al. 2005), and Oulu-CASIA (Zhao et al. 2011)) and two in-the-wild datasets (RAF-DB (Li, Deng, and Du 2017) and SFEW (Dhall et al. 2011)), as multiple source domains for training.”

“To demonstrate the effectiveness of the proposed SC-PFER model, we have performed extensive experiments on two well-known facial expression databases: 1) one 3D facial expression database: BU-3DFE [Yin et al. 2006]; and 2) a facial expression in the wild database: Static Facial Expressions in the Wild (SFEW) [Dhall et…”

“Related databases include FER-Wild [35], FER2013 [36], RAF-DB [37], EmotioNet [38], AffectNet [13], Static Facial Expression in the Wild (SFEW) [39] and Acted Facial Expression in the Wild (AFEW) [40].”

“SFEW database [31] is a part of a temporal facial expressions database acted facial expressions in the wild, which we extracted from movies.”

“frequently yawning, inability to keep eyes open, and swaying the head forward) the VGG-16 architecture is pre-trained for facial expression recognition using the Static Facial Expressions in the Wild (SFEW) database [20].”

“[19] learned a multitask and multi-domain ResNet [10] on both discrete emotion classes (SFEW [4]) and Action Units (EmotioNet [8]).”

“We combine five publicly available labeled facial expression databases to create the HED: (a) Static Facial Expressions in the Wild (SFEW) database [12], (b) Extended Cohn-Kanade database (CK+) [24], (c) MMI database [33], (d) Karolinska Directed Emotional Faces (KDEF) [25], and (e) Denver Intensity of Spontaneous Fac…”

“0 data [7] contains selected frames from different videos of the Acted Facial Expressions in the Wild (AFEW) dataset [6] assigned to either: 225 “angry”, 75 “disgust”, 124 “fear”, 256 “happy”, 228 “neutral”, 234 “sad” and 150 “surprise”.”

“We conduct extensive experiments on SFEW 2.0, AFEW 4.0 and 5.0 (audio part) databases, which were used in Emotion Recognition in the Wild Challenge (EmotiW 2014 and 2015) to evaluate our proposed method.”

“0 [12], and ExpW [13].”

“Acted Facial Expressions in the Wild (AFEW) [1], GENKI [2], Happy People Images (HAPPEI) [1] and Static Facial Expressions in the Wild (SFEW) [3] are recent emotion databases representing real-world scenarios.”

“such as posture, identity, and illumination, and have improved recognition performance for many public datasets collected in the laboratory or through various ways such as CK+ [12,13], MMI [14], Oulu-CASIA [15], SFEW/AFEW [16], FERPlus [17], AffectNet [18], EmotioNet [19], and RAF-DB [20,21].”

“Furthermore, classification tasks based on close to real-world such as Static Facial Expressions in the Wild (SFEW) [145] and AFEW datasets is challenging.”

“For classification, we use SVM [25] with RBF kernel as same as that in [23].”

“Then, the reduced feature vectors are classified using ELM-RBF which is faster than RBF kernel-based SVM.”

“The SFEW [4] dataset was created by selecting specific frames from the AFEW [5] dataset.”

“; þ 90 SFEW [112] In the wild, 7 expressions 95 Movie manually 3 viewpoints (e.g., [68], [69]) AN, DI, FE, JO (HA), NE, SA, SU.”

“SFEW [23] contains 958 training samples, 436 validation samples and 372 test samples, each of which is the static frame selected from Acted Facial Expressions in the Wild (AFEW) [24] dataset.”

“In recent years, significant progress has been made towards developing deep learning (DL) based robust FER systems [6–11] along with the development of large-scale datasets [12–16] captured in a real-world unconstrained environment.”

“The Static Facial Expressions in the Wild (SFEW) [14] was created by using this data collection method.”

“Static Facial Expression in Wild (SFEW) 2.0 [94] dataset is a prominent FER dataset.”

“SFEW database [28] is a part of a temporal facial expressions database acted facial expressions in the wild which we have extracted from movies.”

“The Static Facial Expression in the Wild (SFEW 2.0) database [12] is composed of frames extracted from different movies depicting people having seven different emotional states: anger, disgust, fear, happiness, neutrality, sadness, and surprise.”

“0 [53], and RAF-DB [67] datasets collected in the wild were used in our experiments.”

“extended CK+ [1], JAFFE [2], and static expressions in the wild (SFEW) [3], [4].”

“0 [73] Static Facial Expression in Wild Wild Facial 1322 samples 255 anger, 75 disgust, 124 fear, 228 neutral, 256 happiness, 234 sadness and 150 surprise”

“surprise, contempt, and neutral [31], [32], [33].”

“We report the overall accuracy on its testing set, which is consistent with that in SCN [13], RAN [32], ADDL [26], etc. SFEW [59] is created by selecting keyframes from AFEW [60], which contains 958 training images, 436 validation images, and 372 testing images, showing one of six basic and neutral…”

“In recent years, as more and more open databases of facial expression images captured in the wild have been released, such as RAF-DB [5], AffectNet [6], SFEW [7], FER2013 [8], and so on, there have been more studies on FER in the wild.”

“The proposed approach is assessed on both controlled and uncontrolled datasets. facial expressions under challenging conditions [24].”

“A significant amount of work has focused on employing transfer learning techniques with CNN models such as AlexNet [69], SqueezeNet [70], and VGG19, evaluating their efficacy on benchmark datasets including FER2013, JAFFE, KDEF, CK+, SFEW [71], and KMU-FED.”

“In the EmotiW 2015 challenge [8, 11], the AFEW5.”

“SFEW Database: The Static Facial Expression in the Wild (SFEW) database [6] is a static real world facial expression database which consist of images extracted from movies, is highly different from those databases generated in labcontrolled environment.”

“…evaluation of our proposed method in comparison to other FER meth-ods, we conducted benchmark experiments on two distinct datasets: RaFD [23] and KDEF [24] for evaluating FER results in the controlled laboratory environment, SFEW [25] and RAF-DB [26] for evaluating FER results in the wild.”

“Table 12 shows a comparison of the latest methods and models on facial emotion recognition with the SFEW open dataset to verify the universality and superiority of the designed method.”

“SFEW [63] is a facial expression dataset created by selecting static frames”

“0 % obtained using combined LPQ and PHOG features and a non-linear SVM classifier in [18].”