Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

While most of the research focused on primary FEs and AUs, analysis of pain, fatigue, frustration or cognitive states paved the way to new applications in AFER.

“While most of the research focused on primary FEs and AUs, analysis of pain, fatigue, frustration or cognitive states paved the way to new applications in AFER.”

Classified as probably not a substantial claim.

In order to deal with challenges raised by large pose variations, diversity in illumination conditions and detection of subtle facial behaviour, alternative modalities like 3D and Thermal have been proposed.

“In order to deal with challenges raised by large pose variations, diversity in illumination conditions and detection of subtle facial behaviour, alternative modalities like 3D and Thermal have been proposed.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

From RGB static representations of posed FEs, approaches advanced towards dynamic representations and spontaneous expressions.

“From RGB static representations of posed FEs, approaches advanced towards dynamic representations and spontaneous expressions.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Research in automatic AFER started at the end of the 1970's, but for more than a decade progress was slow mainly because of limitations of face detection and face registration algorithms and lack of sufficient computational power.

“In summary, research in automatic AFER started at the end of the 1970’s, but for more than a decade progress was slow mainly because of limitations of face detection and face registration algorithms and lack of sufficient computational power.”

May combine two claims.

Most of AFER was used for predicting primary affective states of basic emotions, but FEs were also used for predicting non-primary affective states such as complex mental states, fatigue, frustration, pain, depression, mood and personality traits.

“Most of AFER was used for predicting primary affective states of basic emotions, such as anger or happiness but FEs were also used for predicting non-primary affective states such as complex mental states [223], fatigue [224], frustration [44], pain [185], [186], [225], depression [53], [250], mood and personality traits [251], [252].”

• cites [223] El Kaliouby R and Robinson P, “ Real-time inference of complex mental states from facial expressions and head gestures,” in Realtime vision for human-computer interaction. Springer, 2005, pp. 181– 200. sourceNot checked yet

  Not checked in the free run.

• cites [224] Ji Q, Lan P, and Looney C, “ A probabilistic framework for modeling and real-time monitoring human fatigue,” SMC-A, vol. 36, no. 5, pp. 862– 875, 2006. sourceNot checked yet

  Not checked in the free run.

• cites [44] Kapoor A, Burleson W, and Picard RW, “ Automatic prediction of frustration,” IJHCS, vol. 65, no. 8, pp. 724– 736, 2007. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Estimating facial AUs intensity followed a few distinct approaches: critical analysis of limitations of estimating intensity from classification scores, direct estimation from features, studies on optimal representations, and modelling correlations between AUs appearance and intensity priors.

“In summary, estimating facial AUs intensity followed a few distinct approaches. First, some researchers made a critical analysis about the limitations of estimating intensity from classification scores [235], [236], [237]. As an alternative, direct estimation from features was analyzed. Further studies on optimal representations for intensity estimation of different AUs were published either from”

• cites [235] Savran A, Sankur B, and Taha Bilge M, “ Estimation of facial action intensities on 2d and 3D data,” in EUSIPCO, 2011, pp. 1969– 1973. sourceNot checked yet

  Not checked in the free run.

• cites [236] Savran A, Sankur B, and Bilge MT, “ Regression-based intensity estimation of facial action units,” IVC, vol. 30, pp. 774– 784, 2012. sourceNot checked yet

  Not checked in the free run.

• cites [237] Girard JM, Cohn JF, and De la Torre F, “ Estimating smile intensity: A better way,” PRL, 2014. PMC4598946 26461205 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Even critics of strong evolutionary accounts find that recognition of FEs of emotion are universally above chance and in many cases quite higher.

“Even critics of strong evolutionary accounts [31], [32] find that recognition of FEs of emotion are universally above chance and in many cases quite higher.”

• cites [31] Russell JA, “ Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies,” Psychol. Bull, vol. 115, no. 1, pp. 102– 141, 1994. 8202574 sourceNot checked yet

  Not checked in the free run.

• cites [32] Jack RE, Blais C, Scheepers C, Schyns PG, and Caldara R, “ Cultural confusions show that facial expressions are not universal,” Current Biology, vol. 19, pp. 1– 6, 2009. 19682907 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The universal primary emotions hypothesis has been extensively exploited in affective computing.

“Mainly because of its simplicity and its universality claim, the universal primary emotions hypothesis has been extensively exploited in affective computing.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Darwin proposed that FEs are the residual actions of more complete behavioral responses to environmental challenges.

“Darwin proposed that FEs are the residual actions of more complete behavioral responses to environmental challenges.”

May combine two claims.

Strong evidence exists in support of evolutionary accounts of FE, including universality of FEs of emotion, physiological specificity of emotion, and automatic appraisal and unbidden occurrence.

“While some aspects of evolutionary accounts of FE are controversial [20], strong evidence exists in their support. Evidence includes universality of FEs of emotion, physiological specificity of emotion, and automatic appraisal and unbidden occurrence [21], [22], [23].”

• cites [21] Eibl-Eibesfeldt I, “ An argument for basic emotions,” in Cogn. Emot, 1992, pp. 169– 200.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [22] Keltner D and Ekman P, “ Facial expression of emotion,” in Handbook of emotions, 2nd ed., 2000, pp. 236– 249.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [23] Matsumoto D, Keltner D, Shiota MN, O’Sullivan M, and Frank M, “ Facial expressions of emotion,” in Handbook of Emotions, 2008, ch. 13, pp. 211– 234.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A dominant view upon affect is based on the underlying assumption that humans universally express a set of discrete primary emotions which include happiness, sadness, fear, anger, disgust, and surprise.

“More recently, influenced by the research of Paul Ekman [7], [13] a dominant view upon affect is based on the underlying assumption that humans universally express a set of discrete primary emotions which include happiness, sadness, fear, anger, disgust, and surprise”

• cites [7] Ekman P, “ Universal and cultural differences in facial expression of emotion,” Nebr. Sym. Motiv, vol. 19, pp. 207– 283, 1971.Not checked yet

  Not checked in the free run.

• cites [13] Ekman P, “ Strong evidence for universals in facial expressions: A reply to Russell’s mistaken critique,” Psychol. Bull, vol. 115, no. 2, pp. 268– 287, 1994. 8165272Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Due to higher dimensionality, dimensional descriptions can potentially describe more complex and subtle emotions.

“Due to the higher dimensionality of such descriptions they can potentially describe more complex and subtle emotions.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Attempts to describe human emotion mainly fall into two approaches: categorical and dimensional description.

“Attempts to describe human emotion mainly fall into two approaches: categorical and dimensional description.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Another popular approach is to place a particular emotion into a space having a limited set of dimensions including valence, activation, and control.

“Another popular approach is to place a particular emotion into a space having a limited set of dimensions [15], [16], [17]. These dimensions include valence (how pleasent or unpleasent a feeling is) activation2 (how likely is the person to take action under the emotional state) and control (the sense of control over the emotion).”

• cites [15] Greenwald M, Cook E, and Lang P, “ Affective judgment and psychophysiological response: Dimensional covariation in the evaluation of pictorial stimuli,” J. Psychophysiology, no. 3, pp. 51– 64, 1989.Not checked yet

  Not checked in the free run.

• cites [16] Russell J and Mehrabian A, “ Evidence for a three-factor theory of emotions,” J. Research in Personality, vol. 11, pp. 273– 294, 1977. sourceNot checked yet

  Not checked in the free run.

• cites [17] Watson D, Clark LA, and Tellegen A, “ Development and validation of brief measures of positive and negative affect: The PANAS scales,” JPSP, vol. 54, pp. 1063– 1070, 1988. 3397865 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

FACS seeks to describe nearly all possible FEs in terms of anatomically-based facial actions.

“Perhaps the most influential, FACS (1978; 2002) seeks to describe nearly all possible FEs in terms of anatomically-based facial actions [171], [172].”

• cites [171] Ekman P and Friesen WV, Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, 1978.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [172] Ekman P, Friesen WV, and Hager JC, Facial Action Coding System: The Manual on CD ROM. A Human Face, 2002.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Early fusion directly exploits correlations between features from different modalities, and is specially useful when sources are synchronous in time.

“Early fusion directly exploits correlations between features from different modalities, and is specially useful when sources are synchronous in time.”

May combine two claims.

Viola&Jones is fast but has problems with occlusions and large pose variations.

“It is based on a cascade of weak classifiers, but while fast, it has problems with occlusions and large pose variations [10].”

• cites [10] Salah AA, Sebe N, and Gevers T, “ Communication and automatic interpretation of affect from facial expressions,” Affective Computing and Interaction: Psychological, Cognitive and Neuroscientific Perspectives, p. 157, 2010.Not checked yet

  Not checked in the free run.

May combine two claims.

The continuous approach to facial expression recognition has the advantages of representing subtly different expressions, mixtures of primary expressions, and unsupervisedly defining expressions through clustering.

“The advantages of this second approach are the ability to represent subtly different expressions, mixtures of primary expressions, and the ability to unsupervisedly define the expressions through clustering.”

May combine two claims.

Left-frontal asymmetry was greater during enjoyment; right frontal asymmetry was greater during disgust.

“Left-frontal asymmetry was greater during enjoyment; right frontal asymmetry was greater during disgust.”

• cites [34] Ekman P, Davidson RJ, and Friesen WV, “ The Duchenne smile: Emotional expression and brain psychology ii,” JPSP, vol. 58, no. 2, pp. 342– 353, 1990. 2319446 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Fusing multiple modalities has the advantage of increased robustness and conveying complementary information.

“Fusing multiple modalities has the advantage of increased robustness and conveying complementary information.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Pre-training is an unsupervised layer-wise training step that allows for much larger, unlabeled datasets to be used.

“These approaches often use pre-training [189], an unsupervised layer-wise training step that allows for much larger, unlabeled datasets to be used.”

• cites [189] Hinton GE, Osindero S, and Teh YW, “ A fast learning algorithm for deep belief nets,” Neural Comput, vol. 18, no. 7, pp. 1527– 1554, 2006. 16764513 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

It should be possible to extract features combining 3D and thermal information, but to the best of our knowledge it has not been attempted.

“It should also be possible to extract features combining 3D and thermal information, but to the best of our knowledge it has not been attempted.”

May combine two claims.

Appearance feature extraction techniques are applicable to both RGB and thermal modalities, but not to 3D data, which does not convey appearance information.

“These feature extraction techniques are applicable to both RGB and thermal modalities, but not to 3D data, which does not convey appearance information.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Depth information is robust to changes in illumination.

“Depth information is robust to changes in illumination, while thermal images convey information related to changes in the blood flow produced by emotions.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Depression recognition from FEs is a very important application in analysis of psychological distress.

“Depression recognition from FEs is a very important application in analysis of psychological distress [53], [54], [55].”

• cites [53] Girard JM, Cohn JF, Mahoor MH, Mavadati SM, Hammal Z, and Rosenwald DP, “ Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses,” IVC, vol. 32, no. 10, pp. 641– 647, 2014. PMC4217695 25378765 sourceNot checked yet

  Not checked in the free run.

• cites [54] Scherer S, Stratou G, Mahmoud M, Boberg J, Gratch J, Rizzo A, and Morency L-P, “ Automatic behavior descriptors for psychological disorder analysis,” in Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on IEEE, 2013, pp. 1– 8. sourceNot checked yet

  Not checked in the free run.

• cites [55] Joshi J, Dhall A, Goecke R, Breakspear M, and Parker G, “ Neural-net classification for spatio-temporal descriptor based depression analysis,” in Pattern Recognition (ICPR), 2012 21st International Conference on. IEEE, 2012, pp. 2634– 2638.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Pain detection is used for monitoring patient progress in clinical settings.

“Pain detection is used for monitoring patient progress in clinical settings [48], [49], [50].”

• cites [48] Lucey P, Cohn JF, Matthews I, Lucey S, Sridharan S, Howlett J, and Prkachin KM, “ Automatically detecting pain in video through facial action units,” SMC-B, vol. 41, no. 3, pp. 664– 674, 2011. PMC6942457 21097382 sourceNot checked yet

  Not checked in the free run.

• cites [49] Kaltwang S, Rudovic O, and Pantic M, “ Continuous pain intensity estimation from facial expressions,” ISVC, pp. 368– 377, 2012. sourceNot checked yet

  Not checked in the free run.

• cites [50] Irani R, Nasrollahi K, Simon MO, Corneanu CA, Escalera S, Bahnsen C, Lundtoft DH, Moeslund TB, Pedersen TL, Klitgaard M-L, “ Spatiotemporal analysis of rgb-dt facial images for multimodal pain level recognition,” CVPR Workshops, 2015. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

FAP is part of the MPEG4 standard and is used for synthesizing FE for animating virtual faces.

“FAP is now part of the MPEG4 standard and is used for synthesizing FE for animating virtual faces.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Viola&Jones is still one of the most used algorithms for RGB face detection.

“For RGB images, Viola&Jones [60] still is one of the most used algorithms [10], [61], [177].”

• cites [10] Salah AA, Sebe N, and Gevers T, “ Communication and automatic interpretation of affect from facial expressions,” Affective Computing and Interaction: Psychological, Cognitive and Neuroscientific Perspectives, p. 157, 2010.Not checked yet

  Not checked in the free run.

• cites [61] Jones M and Viola P, “ Fast multi-view face detection,” MERL, vol. 3, p. 14, 2003.Not checked yet

  Not checked in the free run.

• cites [177] De la Torre F and Cohn J, “ Facial expression analysis,” in Visual Analysis of Humans, 2011, pp. 377– 409.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Facial expression signals emotion, communicative intent, individual differences in personality, and psychiatric and medical status, and helps to regulate social interaction.

“Facial expression signals emotion, communicative intent, individual differences in personality, and psychiatric and medical status, and helps to regulate social interaction.”

May combine two claims.

Joy decreases the blood flow to the nose, while arousal increases it to the nose, periorbital, lips and forehead.

“Joy decreases the blood flow to the nose, while arousal increases it to the nose, periorbital, lips and forehead [198].”

• cites [198] Ioannou S, Gallese V, and Merla A, “ Thermal infrared imaging in psychophysiology: potentialities and limits,” Psychophysiology, vol. 51, no. 10, pp. 951– 963, 2014. 24961292 PMC4286005 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Appearance features are more stable to noise, allowing for the detection of a more complete set of FEs, being particularly important for detecting microexpressions.

“Appearance features are more stable to noise, allowing for the detection of a more complete set of FEs, being particularly important for detecting microexpressions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

MAX is less granular and less comprehensive than FACS.

“Maximally Discriminative Facial Movement Coding System (MAX) [174], another descriptive system, is less granular and less comprehensive.”

• cites [174] Izard CE, Maximally discriminative facial movement coding system (MAX). Instructional Resources Center, University of Delaware, 1983.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Because a single emotion or affect may result in multiple expressions, there is no 1:1 correspondence between what the face does and its emotion label.

“Because a single emotion or affect may result in multiple expressions, there is no 1:1 correspondence between what the face does and its emotion label.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Based on the observation that some AU are better detected using geometrical features and others using appearance ones, it was suggested that a combination of both might increase recognition performance.

“Based on the observation that some AU are better detected using geometrical features and others using appearance ones, it was suggested that a combination of both might increase recognition performance [127], [139], [183].”

• cites [127] Koelstra S, Pantic M, and Patras I, “ A dynamic texture-based approach to recognition of facial actions and their temporal models,” TPAMI, vol. 32, no. 11, pp. 1940– 1954, 2010. 20847386 sourceNot checked yet

  Not checked in the free run.

• cites [139] Tian Y-L, Kanade T, and Cohn JF, “ Recognizing action units for facial expression analysis,” TPAMI, vol. 23, pp. 97– 115, 2001. PMC4157835 25210210 sourceNot checked yet

  Not checked in the free run.

• cites [183] Pantic M and Bartlett M, “ Machine analysis of facial expressions,” in Face Recognition. I-Tech Education and Publishing, 2007, pp. 377– 416. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Modern FE recognition techniques rely on labeled data to learn discriminative patterns for recognition and, in many cases, feature extraction.

“Modern FE recognition techniques rely on labeled data to learn discriminative patterns for recognition and, in many cases, feature extraction.”

Probably not a substantial claim: may not pass S3 (load-bearing).

AAM is one of the most used methods for 2D face registration.

“Active Appearance Models (AAM) [77] is one of the most used methods for 2D face registration.”

• cites [77] Cootes TF, Edwards GJ, and Taylor CJ, “ Active appearance models,” TPAMI, vol. 23, no. 6, pp. 681– 685, 2001. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Multimodal fusion is required when dealing with multiple data modalities, and can be approached in four different ways: direct, early, late and sequential fusion.

“An additional step, multimodal fusion (Section 3.2.5), is required when dealing with multiple data modalities, usually coming from other sources of information such as speech and physiological data. This step can be approached in four different ways, depending on the stage at which it is introduced: direct, early, late and sequential fusion.”

May combine two claims.

Momentary stress increases the periorbital blood flow, while if sustained the blood flow to the forehead increases.

“It has been found that momentary stress increases the periorbital blood flow, while if sustained the blood flow to the forehead increases [197].”

• cites [197] Pavlidis I, Dowdall J, Sun N, Puri C, Fei J, and Garbey M, “ Interacting with human physiology,” CVIU, vol. 108, no. 1, pp. 150– 170, 2007. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Detecting students' frustration can help improve e-learning experiences.

“Detecting students’ frustration can help improve e-learning experiences [44].”

• cites [44] Kapoor A, Burleson W, and Picard RW, “ Automatic prediction of frustration,” IJHCS, vol. 65, no. 8, pp. 724– 736, 2007. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Maximum rule is sensible to high-confidence errors.

“This technique is sensible to high-confidence errors.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Late fusion is usually considered for asynchronous data sources, and can be trained on modality-specific datasets, increasing the amount of available data.

“On the other hand, late fusion is usually considered for asynchronous data sources, and can be trained on modality-specific datasets, increasing the amount of available data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Early fusion forces the classifier/regressor to work with a higher-dimensional feature space, increasing the likelihood of over-fitting.

“However, it forces the classifier/regressor to work with a higher-dimensional feature space, increasing the likelihood of over-fitting.”

Probably not a substantial claim: may not pass S3 (load-bearing).

MMI introduced temporal labeling of onset, apex and offset.

“It also introduced temporal labeling of onset, apex and offset.”

• cites [114] Pantic M and Patras I, “ Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences,” SMC-B, vol. 36, pp. 433– 449, 2006. 16602602 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The most well known 3D datasets are BU-3DFE, Bosphorus, BU-4DFE and BP4D.

“The most well known 3D datasets are BU-3DFE [206], Bosphorus [195] (still images), BU-4DFE [207] (video) and BP4D [38] (video).”

• cites [206] Yin L, Wei X, Sun Y, Wang J, and Rosato MJ, “ A 3D facial expression database for facial behavior research,” in FG, 2006, pp. 211– 216. sourceNot checked yet

  Not checked in the free run.

• cites [195] Savran A, Alyüz N, Dibeklioğlu H, Çeliktutan O, Gökberk B, Sankur B, and Akarun L, “ Bosphorus database for 3D face analysis,” in BIOID, 2008, vol. 5372, pp. 47– 56. sourceNot checked yet

  Not checked in the free run.

• cites [207] Yin L, Chen X, Sun Y, Worm T, and Reale M, “ A highresolution 3D dynamic facial expression database,” in FG, 2008, pp. 1– 6. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Sum rule and Product rule partially offset the single-classifier weakness problem, but the strengths of each individual modality are not considered.

“While these approaches partially offset the single-classifier weakness problem, the strengths of each individual modality are not considered.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In CK+, the number of posed samples was increased by 22% and spontaneous expressions were added.

“In CK+, the number of posed samples was increased by 22% and spontaneous expressions were added.”

• cites [191] Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, and Matthews I, “ The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression,” in CVPR Workshops, 2010, pp. 94– 101.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

One of the first important datasets made public was the Cohn-Kanade (CK), later extended into CK+.

“One of the first important datasets made public was the Cohn-Kanade (CK) [190], later extended into what was called the CK+ [191].”

• cites [190] Kanade T, Cohn JF, and Tian Y, “ Comprehensive database for facial expression analysis,” in FG, 2000, pp. 46– 53.Not checked yet

  Not checked in the free run.

• cites [191] Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, and Matthews I, “ The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression,” in CVPR Workshops, 2010, pp. 94– 101.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

GEMEP-FERA is a subset of the emotion portrayal dataset GEMEP, specially annotated using FACS.

“GEMEP-FERA is a subset of the emotion portrayal dataset GEMEP, specially annotated using FACS.”

Probably not a substantial claim: may not pass S3 (load-bearing).

SEMAINE contains primitive FEs, FACS annotations, labels of cognitive states, laughs, nods and shakes during interactions with artificial agents.

“SEMAINE contains primitive FEs, FACS annotations, labels of cognitive states, laughs, nods and shakes during interactions with artificial agents.”

• cites [205] McKeown G, Valstar M, Cowie R, Pantic M, and Schroder M,¨ “ The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent,” TAC, vol. 3, no. 1, pp. 5– 17, 2012.Not checked yet

  Not checked in the free run.

May combine two claims.

There are few thermal FE datasets, and all of them also include RGB data.

“There are few thermal FE datasets, and all of them also include RGB data.”

May combine two claims.

CK has limited gender, age and ethnic diversity and contains only frontal views with homogeneous illumination.

“It has limited gender, age and ethnic diversity and contains only frontal views with homogeneous illumination.”

• cites [190] Kanade T, Cohn JF, and Tian Y, “ Comprehensive database for facial expression analysis,” in FG, 2000, pp. 46– 53.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In DISFA, participants were recorded while watching a video specially chosen for inducing emotional states and 12 AUs were coded for each video frame on a 0 to 5 intensity scale.

“Participants were recorded while watching a video specially chosen for inducing emotional states and 12 AUs were coded for each video frame on a 0 (not present) to 5 (maximum intensity) scale [202].”

• cites [202] Mavadati SM, Mahoor MH, Bartlett K, Trinh P, and Cohn JF, “ Disfa: A spontaneous facial action intensity database,” TAC, vol. 4, no. 2, pp. 151– 160, 2013.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

BU-4DFE is a high-resolution 3D dynamic FE dataset with video sequences of 100 frames each from 101 subjects, containing only primary expressions.

“BU-4DFE is a high-resolution 3D dynamic FE dataset [207]. Video sequences, having 100 frames each, are captured from 101 subjects. It only contains primary expressions.”

• cites [207] Yin L, Chen X, Sun Y, Worm T, and Reale M, “ A highresolution 3D dynamic facial expression database,” in FG, 2008, pp. 1– 6. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In BU-3DFE, 6 expressions from 100 different subjects are captured on four different intensity levels.

“In BU-3DFE, 6 expression from 100 different subjects are captured on four different intensity levels.”

• cites [206] Yin L, Wei X, Sun Y, Wang J, and Rosato MJ, “ A 3D facial expression database for facial behavior research,” in FG, 2006, pp. 211– 216. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

CASME is an example of a dataset containing microexpressions.

“CASME [201] is an example of a dataset containing microexpressions.”

• cites [201] Yan W-J, Wu Q, Liu Y-J, Wang S-J, and Fu X, “ Casme database: A dataset of spontaneous micro-expressions collected from neutralized faces,” in FG, 2013, pp. 1– 7.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A limitation of most RGB datasets is the lack of intensity labels.

“A limitation of most RGB datasets is the lack of intensity labels.”

Probably not a substantial claim: may not pass S3 (load-bearing).

AMFED contains spontaneous FEs recorded in natural settings over the Internet.

“AMFED contains spontaneous FEs recorded in natural settings over the Internet.”

• cites [204] McDuff D, El Kaliouby R, Senechal T, Amr M, Cohn JF, and Picard R, “ Amfed facial expression dataset: Naturalistic and spontaneous facial expressions collected ”in-the-wild”,” in CVPR Workshops, 2013, pp. 881– 888. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Direct data fusion has the advantage of allowing the extraction of features from a richer data source, but is limited to input data correlated for both spatial and temporal domains.

“This approach has the advantage of allowing the extraction of features from a richer data source, but is limited to input data correlated for both spatial and, if considered, temporal domains.”

May combine two claims.

While previous RGB datasets record FEs in controlled lab environments, AFEW, AMFED and SEMAINE contain faces in naturalistic environments.

“While previous RGB datasets record FEs in controlled lab environments, Acted Facial Expressions In The Wild Database (AFEW) [203], Affectiva-MIT Facial Expression Dataset (AMFED) [204] and SEMAINE [205] contain faces in naturalistic environments.”

• cites [203] Dhall A, Goecke R, Lucey S, and Gedeon T, “ Acted facial expressions in the wild database,” Australian Nat. U, Tech. Rep., 2011.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [204] McDuff D, El Kaliouby R, Senechal T, Amr M, Cohn JF, and Picard R, “ Amfed facial expression dataset: Naturalistic and spontaneous facial expressions collected ”in-the-wild”,” in CVPR Workshops, 2013, pp. 881– 888. sourceNot checked yet

  Not checked in the free run.

• cites [205] McKeown G, Valstar M, Cowie R, Pantic M, and Schroder M,¨ “ The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent,” TAC, vol. 3, no. 1, pp. 5– 17, 2012. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The first version of CK is relatively small, consisting of posed primary FEs.

“The first version is relatively small, consisting of posed primary FEs.”

• cites [190] Kanade T, Cohn JF, and Tian Y, “ Comprehensive database for facial expression analysis,” in FG, 2000, pp. 46– 53.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Weight criterion assigns a confidence to each classifier output, outputting a weighted linear combination of the predictions.

“Weight criterion solves this by assigning a confidence to each classifier output, otputting a weighted linear combination of the predictions [161], [162], [167], [200].”

• cites [161] Gunes H and Piccardi M, “ Affect recognition from face and body: early fusion vs. late fusion,” in ICSMC, 2005, pp. 3437– 3443. sourceNot checked yet

  Not checked in the free run.

• cites [162] Busso C, Deng Z, Yildirim S, Bulut M, Lee CM, Kazemzadeh A, Lee S, Neumann U, and Narayanan S, “ Analysis of emotion recognition using facial expressions, speech and multimodal information,” in ICMI, 2004, pp. 205– 211. sourceNot checked yet

  Not checked in the free run.

• cites [167] Yoshitomi Y, Kim S-I, Kawano T, and Kilazoe T, “ Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face,” in RO-MAN, 2000, pp. 178– 183. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Sequential fusion uses the results of one modality to disambiguate those of another when needed.

“Sequential fusion is a technique that applies the different modality predictions in sequential order. It uses the results of one modality to disambiguate those of another when needed.”

May combine two claims.

Bosphorus has low ethnic diversity but contains a much larger number of expressions, different head poses and deliberate occlusions.

“Bosphorus has low ethnic diversity but it contains a much larger number of expressions, different head poses and deliberate occlusions.”

• cites [195] Savran A, Alyüz N, Dibeklioğlu H, Çeliktutan O, Gökberk B, Sankur B, and Akarun L, “ Bosphorus database for 3D face analysis,” in BIOID, 2008, vol. 5372, pp. 47– 56. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Probabilistic inference models restrict relations between features, reducing the degrees of freedom of the model.

“The advantage of probabilistic inference models is that the relations between features are restricted, reducing the degrees of freedom of the model.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

AFEW has 957 videos extracted from movies, labeled with six primary expressions and additional information about pose, age, and gender of multiple persons in a frame.

“AFEW has 957 videos extracted from movies, labeled with six primary expressions and additional information about pose, age, and gender of multiple persons in a frame.”

• cites [203] Dhall A, Goecke R, Lucey S, and Gedeon T, “ Acted facial expressions in the wild database,” Australian Nat. U, Tech. Rep., 2011.Not checked yet

  Not checked in the free run.

May combine two claims.

Most 3D methods target primary expressions but studies about AUs were published as well.

“Most of them target primary expressions [131] but studies about AUs were published as well [122], [218].”

• cites [131] Wang J, Yin L, Wei X, and Sun Y, “ 3D facial expression recognition based on primitive surface feature distribution,” in CVPR, 2006, pp. 1399– 1406. sourceNot checked yet

  Not checked in the free run.

• cites [122] Sandbach G, Zafeiriou S, and Pantic M, “ Local normal binary patterns for 3D facial action unit detection,” in ICIP, 2012, pp. 1813– 1816. sourceNot checked yet

  Not checked in the free run.

• cites [218] Savran A, Sankur B, and Bilge MT, “ Comparative evaluation of 3D vs. 2d modality for automatic detection of facial action units,” PR, vol. 45, no. 2, pp. 767– 782, 2012. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The first work on AFER was published in 1978 and tracked the motion of landmarks in an image sequence.

“The first work on AFER was published in 1978 [211]. It was tracking the motion of landmarks in an image sequence.”

• cites [211] Suwa M, Sugie N, and Fujimora K, “ A preliminary note on pattern recognition of human emotional expression,” in IJCPR, 1978, pp. 408– 410.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Others focused on appearance representations like Gabor filters, optical flow and LBPs or combinations.

“Others focused on appearance representations like Gabor filters [99], optical flow and LBPs [97] or combinations between the two [139].”

• cites [99] Littlewort G, Bartlett MS, Fasel I, Susskind J, and Movellan J, “ Dynamics of facial expression extracted automatically from video,” in CVPR Workshops, 2004, p. 80. sourceNot checked yet

  Not checked in the free run.

• cites [97] Shan C, Gong S, and McOwan PW, “ Facial expression recognition based on local binary patterns: A comprehensive study,” IVC, vol. 27, no. 6, pp. 803– 816, 2009. sourceNot checked yet

  Not checked in the free run.

• cites [139] Tian Y-L, Kanade T, and Cohn JF, “ Recognizing action units for facial expression analysis,” TPAMI, vol. 23, pp. 97– 115, 2001. PMC4157835 25210210 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Poor face detection and face registration algorithms and limited computational power caused the subject to receive little attention throughout the next decade.

“Mostly because of poor face detection and face registration algorithms and limited computational power, the subject received little attention throughout the next decade.”

May combine two claims.

The first seminal work observed a correlation between a classifier's output margin (distance to SVM hyperplane) and the intensity of the facial expression, but only weakly for spontaneous FEs.

“It observed a correlation between a classifier’s output margin, in this case the distance to the hyperplane of a SVM classification, and the intensity of the facial expression. Unfortunately this was only weakly observered for spontaneous FEs.”

• cites [234] Bartlett MS, Littlewort GC, Frank MG, Lainscsek C, Fasel IR, and Movellan JR, “ Automatic recognition of facial actions in spontaneous expressions,” JMM, vol. 1, no. 6, pp. 22– 35, 2006. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

In 2000, the CK dataset was published marking the beginning of modern AFER.

“In 2000, the CK dataset was published marking the beginning of modern AFER [139].”

• cites [139] Tian Y-L, Kanade T, and Cohn JF, “ Recognizing action units for facial expression analysis,” TPAMI, vol. 23, pp. 97– 115, 2001. PMC4157835 25210210 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The FERA challenge added a special section for intensity estimation.

“As a consequence the Facial Expression and Recognition (FERA) challenge added a special section for intensity estimation [231], [232].”

• cites [231] Valstar MF, Jiang B, Mehu M, Pantic M, and Scherer K, “ The first facial expression recognition and analysis challenge,” in FG, 2011, pp. 921– 926. 22736651 sourceNot checked yet

  Not checked in the free run.

• cites [232] Valstar M, Girard J, Almaev T, McKeown G, Mehu M, Yin L, Pantic M, and Cohn J, “ Fera 2015-second facial expression recognition and analysis challenge,” FG, 2015. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The work of Mase and Pentland and Paul Ekman marked a revival of this research topic at the beginning of the nineties.

“The work of Mase and Pentland and Paul Ekman marked a revival of this research topic at the beginning of the nineties [212], [213].”

• cites [212] and P MK. A., “ Automatic lipreading by optical-flow analysis,” SCJ, vol. 22, 1991.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [213] Ekman P, Huang TS, Sejnowski TJ, and Hager JC, “ Final report to NSF of the planning workshop on facial expression understanding,” Human Interaction Lab, vol. 378, 1993. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

NVIE contains 215 subjects, each displaying six expressions, both spontaneous and posed.

“The Natural Visible and Infrared facial Expression database (NVIE) contains 215 subjects, each displaying six expressions, both spontaneous and posed [210].”

• cites [210] Wang S, Liu Z, Lv S, Lv Y, Wu G, Peng P, Chen F, and Wang X, “ A natural visible and infrared facial expression database for expression recognition and emotion inference,” T. Multimedia, vol. 12, no. 7, pp. 682– 691, 2010. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

KTFE dataset subjects display posed and spontaneous motions, triggered through audiovisual media.

“In the Kotani Thermal Facial Emotion (KTFE) dataset subjects display posed and spontaneous motions, also triggered through audiovisual media [196].”

• cites [196] Nguyen H, Kotani K, Chen F, and Le B, “ A thermal facial emotion database and its analysis,” in PSIVT, 2014, pp. 397– 408. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A number of studies question the validity of estimating intensity from distance to the classification hyperplane.

“A number of studies question the validity of estimating intensity from distance to the classification hyperplane [235], [236], [237].”

• cites [235] Savran A, Sankur B, and Taha Bilge M, “ Estimation of facial action intensities on 2d and 3D data,” in EUSIPCO, 2011, pp. 1969– 1973. sourceNot checked yet

  Not checked in the free run.

• cites [236] Savran A, Sankur B, and Bilge MT, “ Regression-based intensity estimation of facial action units,” IVC, vol. 30, pp. 774– 784, 2012. sourceNot checked yet

  Not checked in the free run.

• cites [237] Girard JM, Cohn JF, and De la Torre F, “ Estimating smile intensity: A better way,” PRL, 2014. PMC4598946 26461205 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The AVEC challenge is dedicated to dimensional prediction of affect and depression level prediction.

“The AVEC, a challenge, is dedicated to dimensional prediction of affect (valance, arousal, dominance) and depression level prediction.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Cohn et al. concluded that specific AUs have higher predictive power for depression than others, suggesting the advantage of using indirect representations for depression prediction.

“They concluded that specific AUs have higher predictive power for depression than others suggesting the advantage of using indirect representations for depression prediction.”

• cites [187] Cohn JF, Kruez TS, Matthews I, Yang Y, Nguyen MH, Padilla MT, Zhou F, and De la Torre F, “ Detecting depression from facial actions and vocal prosody,” in ACII, 2009, pp. 1– 7. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

RGB representation performs better on the upper face while 3D representation performs better on the lower face, and there is an overall improvement if RGB and 3D intensity estimations are fused.

“When evaluated comparatively, RGB representation performs better on the upper face while 3D representation performs better on the lower face and there is an overall improvement if RGB and 3D intensity estimations are fused.”

• cites [236] Savran A, Sankur B, and Bilge MT, “ Regression-based intensity estimation of facial action units,” IVC, vol. 30, pp. 774– 784, 2012. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Studies about personality are usually based on the BigFive personality trait model.

“Usually studies about personality are based on the BigFive personality trait model which is organized along five factors: openness, conscientiousness, extraversion, agreeableness, and neuroticism.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Microexpressions were first reported by Haggard and Issacs in 1966.

“They were first reported by Haggard and Issacs in 1966 [244].”

• cites [244] Haggard EA and Isaacs KS, “ Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy,” in Methods of research in psychotherapy. Springer, 1966, pp. 154– 165. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Microexpressions detection and classification with high recognition rates are reported even at 25fps.

“Microexpressions detection and classification with high recognition rates are reported even at 25fps.”

• cites [247] Pfister T, Li X, Zhao G, and Pietikainen M, “ Recognising¨ spontaneous facial micro-expressions,” in ICCV, 2011, pp. 1449– 1456. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Microexpressions differ from other expressions because of their short duration, subtleness and localization.

“Microexpressions differ from other expessions not only because of their short duration but also because of their subtleness and localization.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Because of their short duration, microexpressions may be better captured at greater than 30 fps.

“Because of their short duration microexpressions may be better captured at greater than 30 fps.”

May combine two claims.

The approach in [49] is static, which would fail to distinguish between eye blink and eye closure, and does not exploit correlations between apparitions of different AUs.

“On the other hand this approach is static, which would fail to distinguish between eye blink and eye closure, and does not exploit the correlations between apparitions of different AUs.”

• cites [49] Kaltwang S, Rudovic O, and Pantic M, “ Continuous pain intensity estimation from facial expressions,” ISVC, pp. 368– 377, 2012. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

[247] proposes the first microexpressions recognition system.

“[247] proposes the first microexpressions recognition system.”

• cites [247] Pfister T, Li X, Zhao G, and Pietikainen M, “ Recognising¨ spontaneous facial micro-expressions,” in ICCV, 2011, pp. 1449– 1456. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Usually a microexpression lasts between 1/25 and 1/3 of a second and has low intensity.

“Usually a microexpression lasts between 1/25 and 1/3 of a second and has low intensity.”

May combine two claims.

Using FEs achieves better results than more basic visual activity measures like gaze activity and overall motion of the head and body, but performance is considerably worse than when estimating personality from audio and especially from prosodic cues.

“They show that using FEs is achieving better results than more basic visual activity measures like gaze activity and overall motion of the head and body; however performance is considerably worse than when estimating personality from audio and especially from prosodic cues.”

• cites [251] Biel J-I, Teijeiro-Mosquera L, and Gatica-Perez D, “ Facetube: predicting personality from facial expressions of emotion in online conversational video,” in ICMI, 2012, pp. 53– 56.Not checked yet

  Not checked in the free run.

May combine two claims.

Approaches dedicated to depression prediction mainly use direct representations from video without detecting primitive FEs or AUs, based on local, dynamic representations of appearance.

“The approaches dedicated to depression prediction are mainly using direct representations from video without detecting primitive FEs or AUs [253], [254], [255], [256]. They are based on local, dynamic representations of appearance (LBP-TOP or variants) for modelling continuous classification problems.”

• cites [253] Williamson JR, Quatieri TF, Helfer BS, Ciccarelli G, and Mehta DD, “ Vocal and facial biomarkers of depression based on motor incoordination and timing,” in AVEC, 2014, pp. 65– 72. sourceNot checked yet

  Not checked in the free run.

• cites [254] Sidorov M and Minker W, “ Emotion recognition and depression diagnosis by acoustic and visual features: A multimodal approach,” in AVEC, 2014, pp. 81– 86. sourceNot checked yet

  Not checked in the free run.

• cites [255] Senoussaoui M, Sarria-Paja M, Santos JF, and Falk TH, “ Model fusion for multimodal depression classification and level detection,” in AVEC, 2014, pp. 57– 63. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Microexpressions are brief FEs that people in high stake situations make when trying to conceal their feelings.

“Microexpressions are brief FEs that people in high stake situations make when trying to conceal their feelings.”

• cites [244] Haggard EA and Isaacs KS, “ Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy,” in Methods of research in psychotherapy. Springer, 1966, pp. 154– 165. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Multimodality is central in depression prediction approaches, either by early fusion or late fusion with audio representations.

“Multimodality is central in such approaches either by applying early fusion [255] or late fusion [256] with audio representations.”

• cites [255] Senoussaoui M, Sarria-Paja M, Santos JF, and Falk TH, “ Model fusion for multimodal depression classification and level detection,” in AVEC, 2014, pp. 57– 63. sourceNot checked yet

  Not checked in the free run.

• cites [256] Jain V, Crowley JL, Dey AK, and Lux A, “ Depression estimation using audiovisual features and fisher vector encoding,” in AVEC, 2014, pp. 87– 91. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Microexpressions are difficult to recognize for an untrained person, and even after extensive training, human accuracies remain low.

“They are difficult to recognize for an untrained person. Even after extensive training, human accuracies remain low, making an automatic system highly useful.”

May combine two claims.

3D is not necessarily better than RGB; 3D shows improvements on some AUs but performance drops on other AUs, both in detection and intensity estimation.

“Experiments show that 3D is not necessarily better than RGB; in fact, while 3D shows improvements on some AUs, it has performance drops on other AUs, both in the detection and intensity estimation problems.”

• cites [235] Savran A, Sankur B, and Taha Bilge M, “ Estimation of facial action intensities on 2d and 3D data,” in EUSIPCO, 2011, pp. 1969– 1973.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Thermal images are not fit to extract geometric information due to the dullness of the captured image.

“While these types of feature extraction methods are common to all modalities, it has been found that thermal images are not fit to extract geometric information due to the dullness of the captured image.”

May combine two claims.

Some approaches to facial expression detection in naturalistic environments use predesigned representations while recent successful approaches learn the best representation or combine predesigned and learned features.

“Some of the approaches use predesigned representations [194], [262], [263], [264], [265] while recent successful approaches learn the best representation [146], [147], [152] or combine predesigned and learned features [108].”

• cites [194] Dhall A, Goecke R, Lucey S, and Gedeon T, “ Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark,” in ICCV Workshops, 2011, pp. 2106– 2112. sourceNot checked yet

  Not checked in the free run.

• cites [262] Gehrig T and Ekenel HK, “ Why is facial expression analysis in the wild challenging?” in ICMI Workshops, 2013, pp. 9– 16. sourceNot checked yet

  Not checked in the free run.

• cites [146] Liu M, Shan S, Wang R, and Chen X, “ Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition,” in CVPR, 2014, pp. 1749– 1756.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Face detection is the most common approach for face localization and is usually treated as a classification problem where a bounding box can either be a face or not.

“Face detection is the most common approach, and is usually treated as a classification problem where a bounding box can either be a face or not.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Predesigned descriptors are very common, although recently deep learning techniques such as CNN and DBN have been used, implicitly learning the relevant features along with the recognition model.

“Predesigned descriptors are very common, although recently deep learning techniques such as CNN and DBN have been used, implicitly learning the relevant features along with the recognition model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Pixel intensity and skin color were relatively benign sources of error in action unit detection in three-person social interactions.

“Pixel intensity and skin color, by contrast, were relatively benign.”

• cites [261] Girard JM, Cohn JF, Sayette MA, Jeni LA, and De la Torre F, “ Spontaneous facial expression can be measured automatically,” Beh. Res. Meth, 2014. PMC4461567 25488104 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The publication of the naturalistic datasets AMFED and AFEW marked an increasing interest in naturalistic environment analysis.

“The publication of two important naturalistic datasets, AMFED and AFEW marked an increasing interest in naturalistic environment analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

While automatically learned techniques cannot be directly classified according to the nature of the described information, predesigned descriptors exploit either the facial appearance, geometry or a combination of both.

“While automatically learned techniques cannot be directly classified according to the nature of the described information, predesigned descriptors exploit either the facial appearance, geometry or a combination of both.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Multiple persons can express facial expressions simultaneously.

“Even more, multiple persons can express FEs simultaneously.”

May combine two claims.

While approaches to facial expression detection in naturalistic environments using static representations exist, dynamic representations are dominant.

“While approaches to FE detection in naturalistic environments using static representations exist [194], [262], dynamic representations are dominant [108], [113], [146], [147], [263], [264].”

• cites [194] Dhall A, Goecke R, Lucey S, and Gedeon T, “ Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark,” in ICCV Workshops, 2011, pp. 2106– 2112. sourceNot checked yet

  Not checked in the free run.

• cites [262] Gehrig T and Ekenel HK, “ Why is facial expression analysis in the wild challenging?” in ICMI Workshops, 2013, pp. 9– 16. sourceNot checked yet

  Not checked in the free run.

• cites [108] Liu M, Wang R, Li S, Shan S, Huang Z, and Chen X, “ Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild,” in ICMI, 2014, pp. 494– 501. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Despite significant advances, automatic facial expression recognition still faces many challenges like large head pose variations, changing illumination contexts and the distinction between facial display of affect and facial display caused by speech.

“Despite significant advances, automatic AFER still faces many challenges like large head pose variations, changing illumination contexts and the distinction between facial display of affect and facial display caused by speech.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Methods for recognizing other affective states have been proposed, namely cognitive states like boredom, confusion, delight, concentration and frustration, positive and negative affect from groups of people, or liking/not-liking of online media for predicting buying behaviour for marketing purposes.

“Methods for recognizing other affective states have also been proposed, namely cognitive states like boredom, confusion, delight, concentration and frustration [266], positive and negative affect from groups of people [267] or liking/not-linking of online media for predicting buying behaviour for marketing purposes [268].”

• cites [266] Bosch N, D’Mello S, Baker R, Ocumpaugh J, Shute V, Ventura M, Wang L, and Zhao W, “ Automatic detection of learningcentered affective states in the wild,” in Proceedings IUI, 2015, pp. 379– 388. sourceNot checked yet

  Not checked in the free run.

• cites [267] Dhall A, Joshi J, Sikka K, Goecke R, and Sebe N, “ The more the merrier: Analysing the affect of a group of people in images,” in FG, 2015. sourceNot checked yet

  Not checked in the free run.

• cites [268] McDuff D, El Kaliouby R, Senechal T, Demirdjian D, and Picard R, “ Automatic measurement of ad preferences from facial responses gathered over the internet,” IVC, vol. 32, no. 10, pp. 630– 640, 2014. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Face registration is not feasible for thermal imaging due to the dullness of the captured images, which limits feature extraction to appearance-based techniques.

“For instance, face registration is not feasible for thermal imaging due to the dullness of the captured images, which in turn limits feature extraction to appearancebased techniques.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Facial expressions are by definition spontaneous, usually have low intensity, can have multiple apexes and can be difficult to distinguish from facial displays of speech.

“FEs are by definition spontaneous, usually have low intensity, can have multiple apexes and can be difficult to distinguish from facial displays of speech.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Even when one manages to build systems that can robustly recognize facial expressions in naturalistic environments, it still remains difficult to interpret their meaning.

“Finally, even when one manages to build systems that can robustly recognize FEs in naturalistic environments, it still remains difficult to interpret their meaning.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Facial expressions in naturalistic environments are unposed and typically of low to moderate intensity and may have multiple apexes.

“FEs in naturalistic environments are unposed and typically of low to moderate intensity and may have multiple apexes (peaks in intensity).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Currently the main challenge is to improve registration algorithms to robustly deal with naturalistic environments.

“Currently the main challenge is to improve registration algorithms to robustly deal with naturalistic environments.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In spontaneous facial expression recognition in controlled environments, dynamic representations improve the ability to distinguish between subtle expressions.

“This follows the tendency in spontaneous FE recognition in controlled environments where dynamic representations improve the ability to distinguish between subtle expressions.”

May combine two claims.

Large head pose and illumination diversity are common in naturalistic environments.

“Large head pose and illumination diversity are common.”

May combine two claims.

While many studies have showed improvements of prediction when combined with physiological or acoustic cues, facial expressions remain marginal in the study of personality trait prediction.

“Practically though, while many studies have showed improvements of prediction when combined with physiological or acoustic cues, FEs remain marginal in the study of personality trait prediction [251], [257], [259], [260].”

• cites [251] Biel J-I, Teijeiro-Mosquera L, and Gatica-Perez D, “ Facetube: predicting personality from facial expressions of emotion in online conversational video,” in ICMI, 2012, pp. 53– 56. sourceNot checked yet

  Not checked in the free run.

• cites [257] Biel J-I, Tsiminaki V, Dines J, and Gatica-Perez D, “ Hi youtube!: Personality impressions and verbal content in social video,” in ICMI, 2013, pp. 119– 126. sourceNot checked yet

  Not checked in the free run.

• cites [259] Batrinca LM, Mana N, Lepri B, Pianesi F, and Sebe N, “ Please, tell me about yourself: automatic personality assessment using short self-presentations,” in ICMI, 2011, pp. 255– 262. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A common pipeline followed by most methods consists of face detection, face registration, feature extraction and recognition itself.

“Regardless of these approaches, a common pipeline has been presented which is followed by most methods, consisting of face detection, face registration, feature extraction and recognition itself.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Many methods exploit information either at a local level, centering on interest regions sometimes defined by action units based on the FACS/FAP coding, or at a global level, using the whole facial region.

“Regardless of their nature, many methods exploit information either at a local level, centering on interest regions sometimes defined by AUs based on the FACS/FAP coding, or at a global level, using the whole facial region.”

May combine two claims.

Most approaches target primitive facial expressions of affect, but others recognize cognitive states, positive and negative affect from groups of people, and liking/not-liking of online media for predicting buying behaviour for marketing purposes.

“Most approaches target primitive FEs of affect, but others recognize cognitive states [266], postive and negative affect from groups of people [267] and liking/not-linking of online media for predicting buying behaviour for marketing purposes [268].”

• cites [266] Bosch N, D’Mello S, Baker R, Ocumpaugh J, Shute V, Ventura M, Wang L, and Zhao W, “ Automatic detection of learningcentered affective states in the wild,” in Proceedings IUI, 2015, pp. 379– 388. sourceNot checked yet

  Not checked in the free run.

• cites [267] Dhall A, Joshi J, Sikka K, Goecke R, and Sebe N, “ The more the merrier: Analysing the affect of a group of people in images,” in FG, 2015. sourceNot checked yet

  Not checked in the free run.

• cites [268] McDuff D, El Kaliouby R, Senechal T, Demirdjian D, and Picard R, “ Automatic measurement of ad preferences from facial responses gathered over the internet,” IVC, vol. 32, no. 10, pp. 630– 640, 2014. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Face detection and alignment is highly challenging in naturalistic environments, but vital for eliminating rigid motion and head pose from facial expressions.

“Face detection and alignment is highly challenging in this context, but vital for eliminating rigid motion and head pose from facial expressions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For face localization, two main families of methods have been found: face detection and face segmentation.

“For face localization, two main families of methods have been found: face detection and face segmentation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Facial expressions were proven to correlate better than visual activity with personality traits.

“First, FEs were proven to correlate better than visual activity with personality traits [187].”

• cites [187] Cohn JF, Kruez TS, Matthews I, Yang Y, Nguyen MH, Padilla MT, Zhou F, and De la Torre F, “ Detecting depression from facial actions and vocal prosody,” in ACII, 2009, pp. 1– 7. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Until recently, automatic facial expression recognition was mostly performed in controlled environments.

“Until recently AFER was mostly performed in controlled environments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

When combining multiple modalities, a fifth fusion step is added to the pipeline.

“When combining multiple modalities, a fifth fusion step is added to the pipeline.”

May combine two claims.

In recent years, a critical analysis has been made about the limitations of estimating action unit intensity from classification scores, and estimation directly from features was analysed.

“In recent years, a critical analysis has been made about the limitations of estimating AUs intensity from classification scores [235], [236], [237] and estimation directly from features were analysed.”

• cites [235] Savran A, Sankur B, and Taha Bilge M, “ Estimation of facial action intensities on 2d and 3D data,” in EUSIPCO, 2011, pp. 1969– 1973. sourceNot checked yet

  Not checked in the free run.

• cites [236] Savran A, Sankur B, and Bilge MT, “ Regression-based intensity estimation of facial action units,” IVC, vol. 30, pp. 774– 784, 2012. sourceNot checked yet

  Not checked in the free run.

• cites [237] Girard JM, Cohn JF, and De la Torre F, “ Estimating smile intensity: A better way,” PRL, 2014. PMC4598946 26461205 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The vast majority of techniques use a multi-class classification model where a set of emotions (usually the six basic emotions defined by Ekman) or mental states are to be detected.

“The vast majority of techniques use a multi-class classification model where a set of emotions (usually the six basic emotions defined by Ekman) or mental states are to be detected.”

May combine two claims.

Four main fusion approaches have been identified: direct, early, late and sequential fusion, in most cases using conventional fusion techniques.

“Four main fusion approaches have been identified: direct, early, late and sequential fusion, in most cases using conventional fusion techniques.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In the RGB case, geometric information is never extracted at the local static level.

“In the RGB case, geometric information is never extracted at the local static level.”

May combine two claims.

Detection and classification of posed and more recently spontaneous microexpressions have been proposed.

“Detection and classification of posed [245], [246], [248] and more recently spontaneous microexpressions [247] have been proposed.”

• cites [245] Shreve M, Godavarthy S, Manohar V, Goldgof D, and Sarkar S, “ Towards macro-and micro-expression spotting in video using strain patterns,” in WACV, 2009, pp. 1– 6. sourceNot checked yet

  Not checked in the free run.

• cites [246] Shreve M, Godavarthy S, Goldgof D, and Sarkar S, “ Macro-and micro-expression spotting in long videos using spatio-temporal strain,” in FG, 2011, pp. 51– 56. sourceNot checked yet

  Not checked in the free run.

• cites [247] Pfister T, Li X, Zhao G, and Pietikainen M, “ Recognising¨ spontaneous facial micro-expressions,” in ICCV, 2011, pp. 1449– 1456. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Capturing depth information has important advantages over traditional RGB datasets: it is more invariant to rotation and illumination and captures more subtle changes on the face.

“Capturing depth information has important advantages over traditional RGB datasets. It is more invariant to rotation and illumination and captures more subtle changes on the face.”

May combine two claims.

Multimodality can enrich the representation space and improve emotion inference, either by using different video sensors (RGB, Depth, Thermal) or by combining facial expressions with other sources such as body pose, audio, language or physiological information.

“Multimodality can enrich the representation space and improve emotion inference [269], [270], either by using different video sensors (RGB, Depth, Thermal) or by combining FEs with other sources such as body pose, audio, language or physiological information (brain signals, cardiovascular acivity etc.).”

• cites [269] Kuncheva LI, Combining Pattern Classifier: Methods and Algorithms. John Wiley & Sons, 2004.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [270] Russell JA, Bachorowski J, and Fernandez-Dols J, “ Facial and vocal expressions of emotion,” vol. 54, pp. 329– 349, 2003. 12415074 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The recognition of facial expression is strongly influenced by the body posture and this becomes more important as the facial expression is more ambiguous.

“[273] shows that the recognition of FE is strongly influenced by the body posture and that this becomes more important as the FE is more ambiguous.”

• cites [273] Van den Stock J, Righart R, and De Gelder B, “ Body expressions influence recognition of emotions in the face and voice,” Emotion, vol. 7, no. 3, pp. 487– 494, 2007. 17683205 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Due to their briefness, subtleness and localization, most methods for detecting microexpressions use local, dynamic, appearance representations extracted from high frequency video.

“Due to their briefness, subtleness and localization, most of the methods for detecting microexpressions use local, dynamic, appearance representations extracted from high frequency video.”

May combine two claims.

Depth information is useful for detecting microexpressions and facilitates recognizing a wider range of expressions, which would be more difficult with RGB alone.

“This is useful for detecting microexpressions and facilitates recognizing a wider range of expressions, which would be more difficult with RGB alone.”

Probably not a substantial claim: may not pass S4 (based).

Recognition approaches infer emotions or mental states based on the extracted facial expression features.

“Recognition approaches infer emotions or mental states based on the extracted FE features.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

Current 2D registration techniques lack the level of precision required to extract useful information from local shape deformations.

“While it should be possible to do so, we hypothesise that current 2D registration techniques lack the level of precision required to extract useful information from local shape deformations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Research suggests that using classifier scores for predicting intensity is conceptually wrong and that intensity levels should be directly learned from the ground truth.

“Research suggests that using classifier scores for predicting intensity is conceptually wrong and that intensity levels should be directly learned from the ground truth [237].”

• cites [237] Girard JM, Cohn JF, and De la Torre F, “ Estimating smile intensity: A better way,” PRL, 2014. PMC4598946 26461205 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Many methods focus on recognizing a limited set of primary emotions (usually 6), mainly due to a lack of more diverse datasets.

“Many methods focus on recognizing a limited set of primary emotions (usually 6) [115], [116], [123], [130], [137], [145], [146]. This is mainly due to a lack of more diverse datasets.”

• cites [115] Sebe N, Lew MS, Sun Y, Cohen I, Gevers T, and Huang TS, “ Authentic facial expression analysis,” IVC, no. 12, pp. 1856– 1863, 2007. sourceNot checked yet

  Not checked in the free run.

• cites [116] Kotsia I and Pitas I, “ Facial expression recognition in image sequences using geometric deformation features and support vector machines,” TIP, vol. 16, pp. 172– 187, 2007. 17283776 sourceNot checked yet

  Not checked in the free run.

• cites [123] Hayat M, Bennamoun M, and El-Sallam AA, “ Clustering of video-patches on grassmannian manifold for facial expression recognition from 3D videos,” in WACV, 2013, pp. 83– 88. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In the case of learned features, dynamic feature extraction has not been attempted.

“In the case of learned features, to the best of our knowledge, dynamic feature extraction has not been attempted.”

May combine two claims.

In naturalistic environments, most approaches target primitive facial expressions of affect, but methods for recognizing cognitive states, positive and negative affect from groups of people, or liking/not-liking of online media for predicting buying behaviour for marketing purposes are also common.

“In naturalistic environments, most approaches target primitive FEs of affect. Methods for recognizing cognitive states [266], positive and negative affect from groups of people [267] or liking/not-linking of online media for predicting buying behaviour for marketing purposes [268] are also common.”

• cites [266] Bosch N, D’Mello S, Baker R, Ocumpaugh J, Shute V, Ventura M, Wang L, and Zhao W, “ Automatic detection of learningcentered affective states in the wild,” in Proceedings IUI, 2015, pp. 379– 388. sourceNot checked yet

  Not checked in the free run.

• cites [267] Dhall A, Joshi J, Sikka K, Goecke R, and Sebe N, “ The more the merrier: Analysing the affect of a group of people in images,” in FG, 2015. sourceNot checked yet

  Not checked in the free run.

• cites [268] McDuff D, El Kaliouby R, Senechal T, Demirdjian D, and Picard R, “ Automatic measurement of ad preferences from facial responses gathered over the internet,” IVC, vol. 32, no. 10, pp. 630– 640, 2014. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Increasing the number of recognized expressions usually follows two main directions: encoding expressions based on FACS action units instead of directly classifying them, and exploiting additional information provided by 3D facial data.

“Increasing the number of recognized expressions usually follows two main directions. First, expressions can be encoded based on FACS AUs [99], [113], [127], [139] instead of directly being classified. This provides a comprehensive coding of FEs without directly making a judgement on their intentionality. Other methods exploit additional information provided by 3D facial data.”

• cites [99] Littlewort G, Bartlett MS, Fasel I, Susskind J, and Movellan J, “ Dynamics of facial expression extracted automatically from video,” in CVPR Workshops, 2004, p. 80. sourceNot checked yet

  Not checked in the free run.

• cites [113] Walecki R, Rudovic O, Pavlovic V, and Pantic M, “ Variablestate latent conditional random fields for facial expression recognition and action unit detection,” FG, pp. 1– 8, 2015. sourceNot checked yet

  Not checked in the free run.

• cites [127] Koelstra S, Pantic M, and Patras I, “ A dynamic texture-based approach to recognition of facial actions and their temporal models,” TPAMI, vol. 32, no. 11, pp. 1940– 1954, 2010. 20847386 sourceNot checked yet

  Not checked in the free run.