Deep learning for human affect recognition: Insights and new developments

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Individual studies often use multiple modalities or apply deep learning in more than one way.

“Note that individual studies often use multiple modalities or apply deep learning in more than one way.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The review discusses the application of deep learning across 141 studies to learn spatial features from images.

“In this section, we discuss the application of deep learning across 141 studies to learn spatial features from images.”

May combine two claims.

Deep spatial features lead to higher accuracies than handcrafted spatial features; out of 103 studies that reported comparisons, 93% support this finding.

“Deep spatial features lead to higher accuracies than handcrafted spatial features. Out of 103 studies that reported comparisons, 93% support this finding.”

May combine two claims.

Deep learning has gained considerable attention in human affect recognition since 2010, being employed in 52% of studies in 2017, with a 119% average year-on-year increase in the number of published studies.

“Deep learning has gained considerable attention in this field since 2010: Up from one to two studies per year, it is being employed in 52% of studies in 2017a 119% average year-on-year increase in the number of published studies.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

Stage 1 of the literature search yielded a total of 950 studies on human affect recognition.

“Stage 1 of our search yielded a total of 950 studies.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For speech emotion recognition, learning temporal feature representations with DNNs is an active research area.

“For SER in particular, learning of temporal feature representations with DNNs is an active research area.”

May combine two claims.

The application of DNNs for facial expression recognition has attracted the most attention in the literature, featured in almost twice as many studies compared to speech emotion recognition, which is featured second most frequently.

“From Table 1, it is apparent that the application of DNNs for FER has attracted the most attention in the literature, featured in almost twice as many studies compared to SER, which is featured second most frequently.”

May combine two claims.

Between 2010 and 2016, the number of studies on human affect recognition increased overall, with an average year-on-year increase of 25%.

“Between 2010 and 2016 9 , there was an overall increase in the number of studies on human affect recognition (25% year-on-year increase on average).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Stage 2 of the review focused exclusively on 233 studies that use deep learning for affect recognition.

“In Stage 2, we focus exclusively on the 233 studies found in our review that use deep learning for affect recognition.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Learning joint feature representations for early feature fusion across different modalities is a less active but developing application area of DNNs in human affect recognition.

“A less active, but developing application area of DNNs in human affect recognition is learning joint feature representations for the purpose of early feature fusion across different modalities.”

May combine two claims.

57% of reviewed studies report using dropout for fully-connected layers, and 12% report using it for convolutional layers.

“It is widely used in the reviewed studies-57% report using dropout for fullyconnected layers, and 12% report using it for convolutional layers.”

May combine two claims.

Face cropping is standard practice and reported to increase model accuracy (e.g., from 54% to 72% in [94]).

“Face cropping is standard practice, and reported to increase model accuracy (e.g., from 54% to 72% in [94]; see also [15]).”

• cites [94] [94] A. T. Lopes, E. de Aguiar, A. F. De Souza, and T. OliveiraSantos, 'Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order,' Pattern Recognit. , vol. 61, pp. 610-628, 2017. -Not checked yet

  Not checked in the free run.

• cites [15] [15] J. Li, S. Roy, J. Feng, and T. Sim, 'Happiness level prediction with sequential inputs via multiple regressions,' in Proc. Int. Conf. Multimodal Interact. , 2016, pp. 487-493. -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Most of these studies use additional datasets and transfer learning to avoid overfitting.

“Most of these studies use additional datasets and transfer learning to avoid overfitting.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Most recent models use pre-defined sets of LLDs for spatial modeling.

“Most recent models use pre-defined sets of LLDs for spatial modeling.”

May combine two claims.

Most frequently chosen existing architectures are VGG Net (23 studies) and AlexNet (18 studies).

“Most frequently chosen are VGG Net [80] (23 studies) and AlexNet [36] (18 studies).”

• cites [80] [80] K. Simonyan and A. Zisserman, 'Very deep convolutional networks for large-scale image recognition,' in Proc. Int. Conf. Learn. Represent. , 2014. - sourceNot checked yet

  Not checked in the free run.

• cites [36] [36] A. Krizhevsky, I. Sutskever, and G. E. Hinton, 'Imagenet classification with deep convolutional neural networks,' in Proc. NIPS , vol. 25, 2012, pp. 1097-1105. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Li et al. [125] used a six-layer DNN to learn frame-level features from concatenated MFCCs of a sliding context window, yielding an accuracy improvement of more than 10% over GMMs.

“For example, Li et al. [125] used a six-layer DNN to learn frame-level features from concatenated MFCCs of a sliding context window. In their experiments, this yielded an accuracy improvement of more than 10% over GMMs.”

• cites [125] [125] L. Li, Y. Zhao, D. Jiang, Y. Zhang, F. Wang, I. Gonzalez, E. Valentin, and H. Sahli, 'Hybrid deep neural network-hidden markov model (dnn-hmm) based speech emotion recognition,' in Proc. ACII , 2013, pp. 312-317. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Khorrami et al. [87] reported an increase in accuracy of 2.5% after applying dropout to fully-connected layers.

“Khorrami et al. [87] reported an increase in accuracy of 2.5% after applying dropout to fully-connected layers.”

• cites [87] [87] P. Khorrami, T. Paine, and T. Huang, 'Do deep neural networks learn facial action units when doing expression recognition?' in Proc. Int. Conf. Comput. Vision , 2015, pp. 19-27. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Data augmentation artificially increases the number of available examples or training epochs by a factor typically between 10 and 30, and up to 300.

“This way, researchers artificially increase the number of available examples or training epochs by a factor typically between 10 and 30 [88], [113], [114], and up to 300 [115].”

• cites [88] [88] H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, 'Joint fine-tuning in deep neural networks for facial expression recognition,' in Proc. Int. Conf. Comput. Vision , 2015, pp. 2983-2991. - sourceNot checked yet

  Not checked in the free run.

• cites [113] [113] Y. Guo, D. Tao, J. Yu, H. Xiong, Y. Li, and D. Tao, 'Deep neural networks with relativity learning for facial expression recognition,' in Proc. Int. Conf. Multimedia Expo Workshops , 2016, pp. 1-6. - sourceNot checked yet

  Not checked in the free run.

• cites [114] [114] Y. Cai, W. Zheng, T. Zhang, Q. Li, Z. Cui, and J. Ye, 'Video based emotion recognition using cnn and brnn,' in Proc. Chin. Conf. Pattern Recognition , 2016, pp. 679-691. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some studies considered using known architectures from object recognition and regarded it as necessary to pre-train these larger models on ImageNet to avoid overfitting the relatively few labeled examples.

“Some considered using known architectures from object recognition [132], [85], [127]. Here, the authors regarded it as necessary to pre-train these larger models on the ImageNet database to avoid overfitting the relatively few labeled examples.”

• cites [132] [132] C.-T. Ho, Y.-H. Lin, and J.-L. Wu, 'Emotion prediction from usergenerated videos by emotion wheel guided deep learning,' in Proc. Int. Conf. Neural Inf. Process. , 2016, pp. 3-12. - sourceNot checked yet

  Not checked in the free run.

• cites [85] [85] S. Zhang, S. Zhang, T. Huang, W. Gao, and Q. Tian, 'Learning affective features with a hybrid deep model for audio-visual emotion recognition,' IEEE Trans. Circuits Syst. Video Technol. , 2017. - sourceNot checked yet

  Not checked in the free run.

• cites [127] [127] A. M. Badshah, J. Ahmad, N. Rahim, and S. W. Baik, 'Speech emotion recognition from spectrograms with deep convolutional neural network,' in Proc. Int. Conf. Platform Technol. Serv. , 2017, pp. 1-5. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Dropout is applied in fully-connected layers in 43% of studies and in convolutional layers in 14% of studies.

“Dropout is applied in fullyconnected (reported in 43% of studies, e.g., [135]) and convolutional layers (14% of studies, e.g., [108]).”

• cites [135] [135] Y. Zhang, Y. Liu, F. Weninger, and B. Schuller, 'Multi-task deep neural network with shared hidden layers: Breaking down the wall between emotion representations,' in Proc. ICASSP , 2017, pp. 4990-4994. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [108] [108] P. Tzirakis, G. Trigeorgis, M. A. Nicolaou, B. Schuller, and S. Zafeiriou, 'End-to-end multimodal emotion recognition using deep neural networks,' IEEE J. Sel. Top. Signal Process. , vol. 11, no. 8, pp. 1301-1309, 2017. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

All four identified studies achieved accuracy improvements over handcrafted approaches, but a lack of training data was also mentioned.

“All achieved accuracy improvements over handcrafted approaches, but a lack of training data was also mentioned [145], [146].”

• cites [145] [145] M. Yanagimoto and C. Sugimoto, 'Recognition of persisting emotional valence from eeg using convolutional neural networks,' in Proc. Int. Workshop Computat. Intell. Appl. , 2016, pp. 27-32. - sourceNot checked yet

  Not checked in the free run.

• cites [146] [146] T. Zhang, W. Zheng, Z. Cui, Y. Zong, and Y. Li, 'Spatial-temporal recurrent neural network for emotion recognition,' IEEE Trans. Cybern. , 2017. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Zhang et al. [148] and Ranganathan et al. [149] used fully-connected DBN models to learn from aggregated facial feature point trajectories, both improving recognition accuracies over shallow aggregation strategies.

“For example, Zhang et al. [148] and Ranganathan et al. [149] used fully-connected DBN models to learn from aggregated facial feature point trajectories, both improving recognition accuracies over shallow aggregation strategies.”

• cites [148] [148] B. Zhang, G. Essl, and E. Mower Provost, 'Automatic recognition of self-reported and perceived emotion: Does joint modeling help?' in Proc. Int. Conf. Multimodal Interact. , 2016, pp. 217-224. - sourceNot checked yet

  Not checked in the free run.

• cites [149] [149] H. Ranganathan, S. Chakraborty, and S. Panchanathan, 'Multimodal emotion recognition using deep learning architectures,' in Proc. Winter Conf. Appl. Comput. Vision , 2016, pp. 1-9. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Most studies chose custom architectures of one to three convolutional layers and one to three fully-connected layers for this task.

“Most studies chose custom architectures of one to three convolutional layers and one to three fully-connected layers for this task.”

May combine two claims.

Yanagimoto and Sugimoto [145] divided raw 16-channel EEG data into 1s segments and used a seven-layer CNN with 10 ms kernels on the first layer, leading to accuracy improvements of over 20%.

“For example, Yanagimoto and Sugimoto [145] divided the raw 16-channel EEG data into 1s segments and used a seven-layer CNN with 10 ms kernels on the first layer, leading to accuracy improvements of over 20%.”

• cites [145] [145] M. Yanagimoto and C. Sugimoto, 'Recognition of persisting emotional valence from eeg using convolutional neural networks,' in Proc. Int. Workshop Computat. Intell. Appl. , 2016, pp. 27-32. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

SoundNet, a 1D CNN trained with unlabeled video, has been shown to perform well in SER even without fine-tuning, and was featured in a challenge-winning submission.

“SoundNet [138], a 1D CNN trained with unlabeled video, has been shown to perform well in SER even without fine-tuning [139], and was featured in a challenge-winning submission [17].”

• cites [138] [138] Y. Aytar, C. Vondrick, and A. Torralba, 'Soundnet: Learning sound representations from unlabeled video,' in Proc. NIPS , 2016. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [139] [139] S. Pini, O. B. Ahmed, M. Cornia, L. Baraldi, R. Cucchiara, and B. Huet, 'Modeling multimodal cues in a deep learning-based framework for emotion recognition in the wild,' in Proc. Int. Conf. Multimodal Interact. ACM, 2017, pp. 536-543. - sourceNot checked yet

  Not checked in the free run.

• cites [17] [17] S. Chen, Q. Jin, J. Zhao, and S. Wang, 'Multimodal multi-task learning for dimensional and continuous emotion recognition,' in Proc. AVEC , 2017, pp. 19-26. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Bertero et al. [134] proposed one convolutional layer with 25 ms kernels, which achieved a 3% accuracy improvement over LLDs.

“Bertero et al. [134] proposed one convolutional layer with 25 ms kernels, which achieved a 3% accuracy improvement over LLDs.”

• cites [134] [134] D. Bertero and P. Fung, 'A first look into a convolutional neural network for speech emotion detection,' in Proc. ICASSP , 2017, pp. 5115-5119. -Not checked yet

  Not checked in the free run.

May combine two claims.

Trigeorgis et al. [54] were the first to use CNNs for raw waveform in SER; they used two convolutional layers for spatial modeling, almost doubling ground truth correlation over LLDs for arousal, and slightly improving for valence.

“Trigeorgis et al. [54] were the first to do so: They used two convolutional layers for spatial modeling, almost doubling ground truth correlation over LLDs for arousal, and slightly improving for valence.”

• cites [54] [54] G. Trigeorgis, F. Ringeval, R. Brueckner, E. Marchi, M. A. Nicolaou, B. Schuller, and S. Zafeiriou, 'Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network,' in Proc. ICASSP , 2016, pp. 5200-5204. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Without any feature engineering, Mirsamadi et al. were able to learn features similar to LLDs from the raw spectral representation of individual audio frames at 25 ms, leading to an accuracy increase of 4%.

“Without any feature engineering, they were able to learn features similar to LLDs from the raw spectral representation of individual audio frames at 25 ms, leading to an accuracy increase of 4%.”

May combine two claims.

Data augmentation to artificially increase the dataset size is used less often than for FER; notable examples include addition of Gaussian noise, different sampling frequencies, and modified playback speed.

“Data augmentation to artificially increase the dataset size is used less often than for FER; notable examples include the addition of Gaussian noise [129], different sampling frequencies [130], and modified playback speed [142].”

• cites [129] [129] M. W¨ ollmer, M. Kaiser, F. Eyben, B. Schuller, and G. Rigoll, 'Lstm-modeling of continuous emotions in an audiovisual affect recognition framework,' Image Vision Comput. , vol. 31, no. 2, pp. 153-163, 2013. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [130] [130] H. M. Fayek, M. Lech, and L. Cavedon, 'Towards real-time speech emotion recognition using deep neural networks,' in Proc. Int. Conf. Signal Process. Commun. Syst. , 2015, pp. 1-5. - sourceNot checked yet

  Not checked in the free run.

• cites [142] [142] Z. Aldeneh and E. M. Provost, 'Using regional saliency for speech emotion recognition,' in Proc. ICASSP , 2017. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The number of features per utterance varies from several hundred to several thousand, depending on the employed LLDs and HSFs.

“This number varies from several hundred [157] to several thousand [158], [123], depending on the employed LLDs and HSFs.”

• cites [157] [157] B. Schuller, S. Steidl, and A. Batliner, 'The interspeech 2009 emotion challenge,' in Proc. INTERSPEECH , 2009. - sourceNot checked yet

  Not checked in the free run.

• cites [158] [158] B. W. Schuller, S. Steidl, A. Batliner, E. N¨ oth, A. Vinciarelli, F. Burkhardt, R. Van Son, F. Weninger, F. Eyben, T. Bocklet et al. , 'The interspeech 2012 speaker trait challenge,' in Proc. INTERSPEECH , 2012, pp. 254-257. - sourceNot checked yet

  Not checked in the free run.

• cites [123] [123] B. Schuller, S. Steidl, A. Batliner, A. Vinciarelli, K. Scherer, F. Ringeval, M. Chetouani, F. Weninger, F. Eyben, E. Marchi et al. , 'The interspeech 2013 computational paralinguistics challenge,' in Proc. INTERSPEECH , 2013. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Ringeval et al. [65] found that while the given physiological signal has lower predictive power than audiovisual signals, both are complementary.

“They found that while the given physiological signal has lower predictive power than audiovisual signals, both are complementary.”

• cites [65] [65] F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J.-P. Thiran, T. Ebrahimi, D. Lalanne, and B. Schuller, 'Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data,' Pattern Recognit. Lett. , vol. 66, pp. 22-30, 2015. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

As highlighted in Table 3, approach T1 is the primary application of DNNs for feature learning from physiological data.

“As highlighted in Table 3, approach T1 is the primary application of DNNs for feature learning from physiological data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One study suggested that a smaller window size (2s) could be the best choice.

“One study suggested that a smaller window size (2s) could be the best choice [65].”

• cites [65] [65] F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J.-P. Thiran, T. Ebrahimi, D. Lalanne, and B. Schuller, 'Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data,' Pattern Recognit. Lett. , vol. 66, pp. 22-30, 2015. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Since 2016, eight studies have explored combining deep spatial features and RNN-based temporal feature learning.

“Since 2016, eight studies have explored combining deep spatial features and RNN-based temporal feature learning.”

May combine two claims.

Yin et al. [171] used stacked autoencoders to learn high-level representations from various peripheral sensors including skin temperature and blood volume pressure, improving the state-of-the-art by 5%.

“Yin et al. [171] used stacked autoencoders (SAEs) to learn high-level representations from various peripheral sensors including skin temperature and blood volume pressure, improving the state-of-the-art by 5%.”

• cites [171] [171] Z. Yin, M. Zhao, Y. Wang, J. Yang, and J. Zhang, 'Recognition of emotions using multimodal physiological signals and an ensemble deep learning model,' Comput. Meth. Programs Biomed. , vol. 140, pp. 93-110, 2017. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Mirsamadi et al. [63] found that model performance is slightly lower with joint learning of both short-term spatial features and temporal context on the IEMOCAP dataset, while both improve model performance when applied independently.

“Mirsamadi et al. [63] found that model performance is slightly lower with joint learning of both short-term spatial features and temporal context on the IEMOCAP dataset [165], while both improve model performance when applied independently.”

• cites [63] [63] S. Mirsamadi, E. Barsoum, and C. Zhang, 'Automatic speech emotion recognition using recurrent neural networks with local attention,' in Proc. ICASSP , 2017, pp. 2227-2231. - sourceNot checked yet

  Not checked in the free run.

• cites [165] [165] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, 'Iemocap: Interactive emotional dyadic motion capture database,' Language Resources Eval. , vol. 42, no. 4, p. 335, 2008. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Studies aiming to reduce feature space dimensionality with deep learning almost exclusively use fully-connected DNNs, consisting of two to four hidden layers.

“Studies aiming to reduce feature space dimensionality with deep learning almost exclusively use fully-connected DNNs, consisting of two to four hidden layers.”

May combine two claims.

Applications of similar architectures based on the raw waveform have also been attempted, showing that end-to-end learning can outperform shallow models.

“Applications of similar architectures based on the raw waveform have also been attempted [54], [108], showing that end-to-end learning can outperform shallow models.”

• cites [54] [54] G. Trigeorgis, F. Ringeval, R. Brueckner, E. Marchi, M. A. Nicolaou, B. Schuller, and S. Zafeiriou, 'Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network,' in Proc. ICASSP , 2016, pp. 5200-5204. - sourceNot checked yet

  Not checked in the free run.

• cites [108] [108] P. Tzirakis, G. Trigeorgis, M. A. Nicolaou, B. Schuller, and S. Zafeiriou, 'End-to-end multimodal emotion recognition using deep neural networks,' IEEE J. Sel. Top. Signal Process. , vol. 11, no. 8, pp. 1301-1309, 2017. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

DNNs are applied for learning of spatial feature representations, temporal feature representations, and joint feature representations for multimodal data.

“They are applied for learning of (i) spatial feature representations, (ii) temporal feature representations, and (iii) joint feature representations for multimodal data.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both AffectNet and EmotioNet are large web-based databases, each at around 1M labeled images.

“Both AffectNet and EmotioNet are large web-based databases, each at around 1M labeled images.”

May combine two claims.

Deep temporal features lead to higher accuracies than handcrafted temporal features, with 92% of 73 studies that reported comparisons supporting this finding.

“Deep temporal features lead to higher accuracies than handcrafted temporal features. Out of 73 studies that reported comparisons, 92% support this finding.”

May combine two claims.

Overall, 190 studies use categorical models (82%), 37 studies use dimensional models (16%), and 6 use both (2.6%).

“Overall, 190 studies use categorical models (82%), 37 studies use dimensional models (16%), and only 6 use both (2.6%).”

May combine two claims.

Of the 77 public databases used in the reviewed studies, 58 use categorical models (75%), 14 use dimensional models (18%), and 4 use both (5.2%).

“Of the 77 public databases used in the studies covered in our review, 58 use categorical models (75%), 14 use dimensional models (18%), and only 4 use both (5.2%).”

May combine two claims.

To date, there is no consensus whether feature-level fusion with deep learning leads to superior accuracy over simple decision fusion, with only 69% of 16 studies that reported comparisons finding that it does.

“To date, there is no consensus whether feature-level fusion with deep learning leads to superior accuracy over simple decision fusion. Out of 16 studies that reported comparisons, only 69% find that it does.”

Probably not a substantial claim: may not pass S3 (load-bearing).

AffectNet includes both dimensional and categorical labels.

“Notably, AffectNet includes both dimensional and categorical labels, encouraging studies to bridge the gap between affect representations.”

May combine two claims.

The AUTOENCODER dataset is the largest face video dataset with 6.5M examples; it contains only 2777 labeled examples and is thus largely unlabeled.

“The AUTOENCODER dataset is the largest face video dataset with 6.5M examples; the dataset contains only 2777 labeled examples and is thus largely unlabeled.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Since 2015, there are studies using deep learning for both spatial and temporal feature learning.

“Since 2015, there are studies using deep learning for both spatial and temporal feature learning.”

May combine two claims.

At this stage, the number of examples per dataset does not show a clear upward trend, and only few deep learning studies covered in the review investigate both categorical and dimensional models.

“Unfortunately, at this stage, the number of examples per dataset does not see a clear upward trend yet and only few deep learning studies covered in our review investigate both types of models.”

May combine two claims.

Compared to older databases, databases published in 2016 and 2017 contain many more examples, made possible by (semi-)automating the labeling process or providing unlabeled examples.

“Compared to older databases, they contain many more examples, which is made possible by (semi-)automating the labeling process, or providing unlabeled examples.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The visual modality is featured most frequently in the databases.

“As expected from previous findings, the visual modality is featured most frequently.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Unlabeled databases are used exclusively for transfer learning.

“Unlabeled databases are used exclusively for transfer learning.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The review reports on the approaches of 21 studies that use deep learning for joint feature learning from multimodal data.

“Here, we report on the approaches of 21 studies that use deep learning for joint feature learning from multimodal data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

More recent databases such as FER2013 and SFEW2 tend to contain more examples than older databases such as JAFFE and CK+.

“Here, more recent databases such as FER2013 and SFEW2 tend to contain more examples than older databases such as JAFFE and CK+-this is made possible by resorting to sources like the web and semi-automatic labeling procedures as opposed to manual annotation of data collected in a laboratory.”

May combine two claims.

Dimensional models of affect are used in 10% of spatial feature learning studies, 30% of temporal feature learning studies, and 52% of studies on joint feature learning.

“While dimensional models of affect are only used in 10% of spatial and 30% of temporal feature learning studies, they are employed in 52% of studies on joint feature learning.”

May combine two claims.

Dimensional models are already employed in 52% of joint feature learning studies and 30% of temporal feature learning studies.

“This already becomes apparent in joint feature learning and to some extent in temporal feature learning, where dimensional models are already employed in 52% and 30% of the reviewed studies, respectively.”

May combine two claims.

The majority of studies still rely on categorical models of affect (82%), particularly in spatial feature learning (92%).

“As can be seen in our review, the majority of studies still rely on categorical models of affect (82%), particularly in spatial feature learning (92%).”

May combine two claims.

While handcrafted features are typically outperformed by DNN-learned features in direct comparison, some challenge-winning models rely on hybrid architectures that take advantage of complementarities between the two.

“While they are typically outperformed by DNN-learned features in direct comparison, some challenge-winning models rely on hybrid architectures that can take advantage of complementarities between the two [152], [18].”

• cites [152] [152] J. Yan, W. Zheng, Z. Cui, C. Tang, T. Zhang, Y. Zong, and N. Sun, 'Multi-clue fusion for emotion recognition in the wild,' in Proc. Int. Conf. Multimodal Interact. , 2016, pp. 458-463. - sourceNot checked yet

  Not checked in the free run.

• cites [18] [18] K. Brady, Y. Gwon, P. Khorrami, E. Godoy, W. Campbell, C. Dagli, and T. S. Huang, 'Multi-modal audio, video and physiological sensor learning for continuous emotion prediction,' in Proc. AVEC , 2016, pp. 97-104. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Mehrabian posited that 55% of the emotion conveyed in a message is perceived visually.

“Mehrabian [71] famously posited that 55% of the emotion conveyed in a message is perceived visually.”

• cites [71] [71] A. Mehrabian, 'Communication without words,' Psychology Today , vol. 2, no. 4, pp. 53-56, 1968. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Multimodal sensor combination improves recognition accuracy and robustness.

“multimodal sensor combination was found to improve recognition accuracy and robustness [10].”

• cites [10] [10] A. Jaimes and N. Sebe, 'Multimodal human-computer interaction: A survey,' Comput. Vision and Image Understanding , vol. 108, no. 1, pp. 116-134, 2007. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Since 2010, deep learning methods have been applied to affect recognition problems across multiple modalities and have led to improvements in accuracy, including winning performances at EmotiW and AVEC.

“Since 2010, deep learning methods have been applied to affect recognition problems across multiple modalities and led to improvements in accuracy, including winning performances at EmotiW [14], [15], [16] and AVEC [17], [18], [19].”

• cites [14] [14] Y. Fan, X. Lu, D. Li, and Y. Liu, 'Video-based emotion recognition using cnn-rnn and c3d hybrid networks,' in Proc. Int. Conf. Multimodal Interact. , 2016, pp. 445-450. -Not checked yet

  Not checked in the free run.

• cites [15] [15] J. Li, S. Roy, J. Feng, and T. Sim, 'Happiness level prediction with sequential inputs via multiple regressions,' in Proc. Int. Conf. Multimodal Interact. , 2016, pp. 487-493. -Not checked yet

  Not checked in the free run.

• cites [16] [16] B.-K. Kim, H. Lee, J. Roh, and S.-Y. Lee, 'Hierarchical committee of deep cnns with exponentially-weighted decision fusion for static facial expression recognition,' in Proc. Int. Conf. Multimodal Interact. , 2015, pp. 427-434. -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

CNNs are well suited to learn spatial features directly from image pixels.

“CNNs are well suited to learn spatial features directly from image pixels.”

May combine two claims.

Since around 2010, deep learning methods have started to fulfill crucial roles in human affect recognition systems and are currently driving most state-of-the-art results in this field.

“Since around 2010, deep learning methods have started to fulfill crucial roles in human affect recognition systems. Indeed, it is safe to state that they are currently driving most state-of-the-art results in this field.”

May combine two claims.

Research strives towards 'deeper' models to achieve higher accuracy but runs into the problem of overfitting, which is the main challenge for current research.

“To achieve higher accuracy, research strives towards 'deeper' models (Fig. 4 illustrates this for FER), but runs into the problem of overfitting. This is the main challenge for current research.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The full potential of deep learning in affect recognition may not have been seen yet.

“This suggests that the full potential of deep learning in affect recognition may not have been seen yet.”

• AssumptionThe authors state this as an assumption: Deep and handcrafted features are complementary in affect recognition.

May combine two claims.

DNNs are more efficient at learning and representing the given functions, both statistically and regarding human involvement.

“As discussed in Section 2.1, DNNs are more efficient at learning and representing the given functions, both statistically and regarding human involvement [20].”

• cites [20] [20] Y. Bengio and Y. LeCun, 'Scaling learning algorithms towards ai,' in Large-Scale Kernel Machines , L. Bottou, O. Chapelle, D. DeCoste, and J. Weston, Eds. MIT Press, 2007, pp. 1-41. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Shallow architectures may not be the most efficient way to approach challenging learning problems such as affect recognition.

“it has been suggested that shallow architectures may not be the most efficient way to approach challenging learning problems such as affect recognition [20].”

• cites [20] [20] Y. Bengio and Y. LeCun, 'Scaling learning algorithms towards ai,' in Large-Scale Kernel Machines , L. Bottou, O. Chapelle, D. DeCoste, and J. Weston, Eds. MIT Press, 2007, pp. 1-41. -Not checked yet

  Not checked in the free run.

May combine two claims.

Pre-processing steps reduce the amount of variation the model has to account for, and thus simplify the learning task.

“These steps reduce the amount of variation the model has to account for, and thus simplify the learning task.”

May combine two claims.

In transfer learning, the goal is to use additional corpora to learn generic visual descriptors that are found to be effective in improving initial model parameters, and thus reduce overfitting.

“In transfer learning , the goal is to use additional corpora to learn generic visual descriptors that are found to be effective in improving initial model parameters [109], and thus reduce overfitting.”

• cites [109] [109] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, 'Cnn features off-the-shelf: An astounding baseline for recognition,' in Proc. CVPR Workshops , 2014, pp. 806-813. -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

To avoid overfitting, both dropout and batch normalization can help to achieve better regularization.

“To avoid overfitting, both dropout [116] and batch normalization [33] can help to achieve better regularization.”

• cites [116] [116] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, 'Dropout: A simple way to prevent neural networks from overfitting,' J. Mach. Learn. Res. , vol. 15, no. 1, pp. 1929-1958, 2014. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [33] [33] S. Ioffe and C. Szegedy, 'Batch normalization: Accelerating deep network training by reducing internal covariate shift,' in Proc. Mach. Learn. Res. , vol. 37, 2015, pp. 448-456. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Another way to learn spatial features from the EEG signal is to reflect it as a 2D map representing the location of the electrodes on the head.

“Another way to learn spatial features from the EEG signal is to reflect it as a 2D map representing the location of the electrodes on the head [147].”

• cites [147] [147] J. Li, Z. Zhang, and H. He, 'Implementation of eeg emotion recognition system based on hierarchical convolutional neural networks,' in Proc. Int. Conf. Brain Inspired Cogn. Syst. , 2016, pp. 22-33. - sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

RNNs are well-suited to model the temporal variation of frame-level spatial features.

“The properties of RNNs, as discussed in Section 2.1, make them well-suited to model the temporal variation of frame-level spatial features.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Appearance features can be extended for spatio-temporal representation by considering a third orthogonal plane.

“Appearance features can be extended for spatio-temporal representation by considering a third orthogonal plane [96], [99].”

• cites [96] [96] M. M. Ghazi and H. K. Ekenel, 'Automatic emotion recognition in the wild using an ensemble of static and dynamic representations,' in Proc. Int. Conf. Multimodal Interact. , 2016, pp. 514-521. - sourceNot checked yet

  Not checked in the free run.

• cites [99] [99] H. Kaya, F. G¨ urpınar, and A. A. Salah, 'Video-based emotion recognition in the wild using deep transfer learning and score fusion,' Image Vision Comput. , vol. 65, pp. 66-75, 2017. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Mirsamadi et al. [63] pointed out that most commonly used frame-level LLDs in SER can be derived from spectral representations of the raw speech signal.

“Mirsamadi et al. [63] pointed out that most commonly used frame-level LLDs in SER can be derived from spectral representations of the raw speech signal.”

• cites [63] [63] S. Mirsamadi, E. Barsoum, and C. Zhang, 'Automatic speech emotion recognition using recurrent neural networks with local attention,' in Proc. ICASSP , 2017, pp. 2227-2231. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Semi-supervised learning can give access to knowledge contained in unlabeled datasets.

“Semi-supervised learning can give access to knowledge contained in unlabeled datasets [140].”

• cites [140] [140] J. Deng, X. Xu, Z. Zhang, S. Fr¨ uhholz, and B. Schuller, 'Universum autoencoder-based domain adaptation for speech emotion recognition,' IEEE Signal Process. Lett. , vol. 24, no. 4, pp. 500-504, 2017. -Not checked yet

  Not checked in the free run.

May combine two claims.

The RNN approach allows processing long sequences and both sequence-level and continuous frame-level affect recognition on image sequences of arbitrary length.

“Advantages of this approach include the ability to process long sequences, and the possibility of both sequence-level and continuous frame-level affect recognition on image sequences of arbitrary length.”

May combine two claims.

Since there is no consensus in the literature over a 'universal' handcrafted feature set with superior performance, many recent studies have applied a 'bruteforce' approach, resulting in a large number of features per utterance.

“Since there is no consensus in the literature over a 'universal' handcrafted feature set with superior performance [121], many recent studies have applied a 'bruteforce' approach, resulting in a large number of features per utterance.”

• cites [121] [121] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. Andr´ e, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan et al. , 'The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,' IEEE Trans. Affect. Comput. , vol. 7, no. 2, pp. 190-202, 2016. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Slow fusion treats the time domain like a spatial domain, progressively learning low-level to high-level temporal features, and effectively reduces the amount of parameters required due to temporal depth, allowing for more input frames.

“For example, Gupta et al. [62] used a variant called slow fusion [153], which treats the time domain like a spatial domain, progressively learning low-level to highlevel temporal features. As the amount of parameters required due to the temporal depth of the input is effectively reduced, this allows for more input frames.”

• cites [62] [62] O. Gupta, D. Raviv, and R. Raskar, 'Multi-velocity neural networks for facial expression recognition in videos,' IEEE Trans. Affect. Comput. , 2017. - sourceNot checked yet

  Not checked in the free run.

• cites [153] [153] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, 'Large-scale video classification with convolutional neural networks,' in Proc. CVPR , 2014, pp. 1725-1732. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

When using 3D CNN for spatio-temporal modeling of image sequences, the line between spatial and temporal representation learning can be blurred.

“When using 3D CNN for spatio-temporal modeling of image sequences as discussed in Section 3.1.1, the line between spatial and temporal representation learning can be blurred.”

May combine two claims.

End-to-end training of the entire CNN-LSTM system addresses the disconnected training problem and can lead to accuracy improvements.

“End-to-end training of the entire CNN-LSTM system addresses this problem, and can lead to accuracy improvements [108].”

• cites [108] [108] P. Tzirakis, G. Trigeorgis, M. A. Nicolaou, B. Schuller, and S. Zafeiriou, 'End-to-end multimodal emotion recognition using deep neural networks,' IEEE J. Sel. Top. Signal Process. , vol. 11, no. 8, pp. 1301-1309, 2017. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

It is generally accepted in the literature that multimodal sensor combinations have complementary effects and thus may increase model accuracy.

“It is generally accepted in the literature that multimodal (e.g., audiovisual) sensor combinations have complementary effects and thus may increase model accuracy [11].”

• cites [11] [11] Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, 'A survey of affect recognition methods: Audio, visual, and spontaneous expressions,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 31, no. 1, pp. 39-58, 2009. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Ringeval et al. concluded that in feature-level fusion, ECG data helps for prediction of valence, but not arousal.

“They concluded that in featurelevel fusion, ECG data helps for prediction of valence, but not arousal.”

• cites [65] [65] F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J.-P. Thiran, T. Ebrahimi, D. Lalanne, and B. Schuller, 'Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data,' Pattern Recognit. Lett. , vol. 66, pp. 22-30, 2015. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The ambiguity of human affect inherently makes both affect recognition and the labeling process difficult, and there is an accuracy limit in the degree of agreement between multiple labelers.

“The ambiguity of human affect inherently makes both affect recognition and the labeling process difficult-there is an accuracy limit in the degree of agreement between multiple labelers.”

May combine two claims.

Ranganathan et al. demonstrated the feasibility of learning joint feature representations of AVP sensor data with approach J1 and a DBN, but did not compare performances of different modality combinations.

“Ranganathan et al. [149] demonstrated the feasibility of learning joint feature representations of AVP sensor data with approach J1 and a DBN, but do not compare the performances of different modality combinations.”

• cites [149] [149] H. Ranganathan, S. Chakraborty, and S. Panchanathan, 'Multimodal emotion recognition using deep learning architectures,' in Proc. Winter Conf. Appl. Comput. Vision , 2016, pp. 1-9. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The higher difficulty and inherent ambiguity of affective displays, unreliable ground truth, and imbalanced datasets contribute to the limited impact of deep learning in affect recognition.

“This can in part be attributed to the higher difficulty and inherent ambiguity of affective displaysalthough many databases now provide annotations from multiple labelers, the ground truth tends to be unreliable and datasets are often imbalanced.”

May combine two claims.

Tzirakis et al. used a two-layer LSTM network for feature fusion on raw audiovisual data with two CNNs, and it outperformed the state of the art.

“Learning from raw audiovisual data with two CNNs, Tzirakis et al. [108] used a two-layer LSTM network for feature fusion, which was found to outperform the state of the art.”

• cites [108] [108] P. Tzirakis, G. Trigeorgis, M. A. Nicolaou, B. Schuller, and S. Zafeiriou, 'End-to-end multimodal emotion recognition using deep neural networks,' IEEE J. Sel. Top. Signal Process. , vol. 11, no. 8, pp. 1301-1309, 2017. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Both Yin et al. and Liu et al. found that the representation learned through feature-level fusion leads to improved accuracy over individual modalities.

“Both studies found that the representation learned through feature-level fusion leads to improved accuracy over individual modalities.”

• cites [171] [171] Z. Yin, M. Zhao, Y. Wang, J. Yang, and J. Zhang, 'Recognition of emotions using multimodal physiological signals and an ensemble deep learning model,' Comput. Meth. Programs Biomed. , vol. 140, pp. 93-110, 2017.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [175] [175] W. Liu, W.-L. Zheng, and B.-L. Lu, 'Emotion recognition using multimodal deep learning,' in Proc. Int. Conf. Neural Inf. Process. , 2016, pp. 521-529. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Zhang et al. used a DBN to fuse frame-level audiovisual features learned independently via CNNs, and the learned features are average-pooled for classification at the video level and lead to an improvement over state-of-the-art methods.

“Zhang et al. [85] used a DBN to fuse frame-level audiovisual features learned independently via CNNs; the learned features are averagepooled for classification at the video level and lead to an improvement over state-of-the-art methods.”

• cites [85] [85] S. Zhang, S. Zhang, T. Huang, W. Gao, and Q. Tian, 'Learning affective features with a hybrid deep model for audio-visual emotion recognition,' IEEE Trans. Circuits Syst. Video Technol. , 2017. -Not checked yet

  Not checked in the free run.

The quote only partly states this claim.

Learning from intermediate representations based on differential entropy also outperforms shallow models.

“Similar results are reported when learning from intermediate representations based on differential entropy [55].”

• cites [55] [55] X. Li, D. Song, P. Zhang, G. Yu, Y. Hou, and B. Hu, 'Emotion recognition from multi-channel eeg data through convolutional recurrent neural network,' in Proc. Int. Conf. Bioinf. and Biomed. , 2016, pp. 352-359. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Yanagimoto and Sugimoto used a CNN on raw 16-channel EEG data to differentiate between positive and negative affective states, and this approach outperformed shallow models based on common features.

“Yanagimoto and Sugimoto [145] used a CNN on raw 16-channel EEG data to differentiate between positive and negative affective states, which is shown to outperform shallow models based on common features.”

• cites [145] [145] M. Yanagimoto and C. Sugimoto, 'Recognition of persisting emotional valence from eeg using convolutional neural networks,' in Proc. Int. Workshop Computat. Intell. Appl. , 2016, pp. 27-32. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The small size of labeled datasets hinders the ability of deep models to generalize well and makes it difficult to train large models.

“It hinders the ability of deep models to generalize well, and makes it difficult to train large models.”

May combine two claims.

Gupta et al. demonstrated the feasibility of semi-supervised learning for affective computing, achieving promising results using 2777 labeled face videos to learn from the AUTOENCODER dataset of 6.5M unlabeled face video clips.

“Gupta et al. [62] demonstrated the feasibility of semi-supervised learning for affective computing, achieving promising results using 2777 labeled face videos to learn from the AUTOENCODER dataset of 6.5M unlabeled face video clips.”

• cites [62] [62] O. Gupta, D. Raviv, and R. Raskar, 'Multi-velocity neural networks for facial expression recognition in videos,' IEEE Trans. Affect. Comput. , 2017. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Most of these approaches train individual components separately, which may lead to suboptimal performance considering the whole system.

“However, most of these approaches train individual components separately (e.g., [106], [90]), which may lead to suboptimal performance considering the whole system.”

• cites [106] [106] S. E. Kahou, V. Michalski, K. Konda, R. Memisevic, and C. Pal, 'Recurrent neural networks for emotion recognition in video,' in Proc. Int. Conf. Multimodal Interact. , 2015, pp. 467-474. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [90] [90] P. Khorrami, T. Le Paine, K. Brady, C. Dagli, and T. Huang, 'How deep neural networks can improve emotion recognition on video data,' in Proc. Int. Conf. Image Process. , 2016, pp. 619-623. - sourceNot checked yet

  Not checked in the free run.

May combine two claims.

In practice, specific details of available modalities and requirements regarding predictions are relevant factors, and available labels on common databases widely dictate which representation is used.

“In practice, specific details of available modalities (e.g., static images or video data) and requirements regarding predictions (e.g., continuous or video-level) are relevant factors, and available labels on common databases widely dictate which representation is used.”

May combine two claims.

Mirsamadi et al. found that even a simple attention mechanism contributes to SER by learning to ignore silent frames, adding 1-2% to model accuracy.

“Mirsamadi et al. [63] found that even a simple attention mechanism contributes to SER by, for example, learning to ignore silent frames-an improvement that is easily interpretable and adds 1-2% to model accuracy.”

• cites [63] [63] S. Mirsamadi, E. Barsoum, and C. Zhang, 'Automatic speech emotion recognition using recurrent neural networks with local attention,' in Proc. ICASSP , 2017, pp. 2227-2231. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Future multimodal affect recognition systems are likely to be trained in an end-to-end fashion, which aligns training of the entire network closer with the true performance measure but requires a larger amount of training examples.

“Future multimodal affect recognition systems are likely to be trained in an end-to-end fashion, which aligns training of the entire network closer with the true performance measure [195] but requires a larger amount of training examples.”

• cites [195] [195] A. Graves and N. Jaitly, 'Towards end-to-end speech recognition with recurrent neural networks,' in Proc. Int. Conf. Mach. Learn. , 2014, pp. 1764-1772.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Handcrafted features are readily available, widely recognized, and well-designed for specific applications such as FER and SER.

“This can be attributed to multiple factors: Handcrafted features are readily available, widely recognized, and well-designed for specific applications such as FER and SER.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Given the nature of sensor data in typical HCI scenarios, dimensional representations and combinations of categorical and dimensional models of affect will become even more relevant as a more natural way of dealing with continuous data.

“However, given the nature of sensor data in typical HCI scenarios, it can be expected that dimensional representations, and combinations of categorical and dimensional models of affect will become even more relevant as a more natural way of dealing with continuous data [53].”

• cites [53] [53] H. Gunes and B. Schuller, 'Categorical and dimensional affect analysis in continuous input: Current trends and future directions,' Image Vision Comput. , vol. 31, pp. 120-136, 2013. - sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

DNNs are often described as 'black boxes', as their inner workings are complex and difficult to interpret.

“In general, DNNs are often described as 'black boxes', as their inner workings are complex and difficult to interpret.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This development suggests that affect recognition too may benefit from much larger datasets.

“This development suggests that affect recognition too may benefit from much larger datasets.”

May combine two claims.

While unsupervised learning allows models to learn better initial parameters from unlabeled datasets, fine-tuning with labeled data is still required to guide the model towards its learning goal.

“While unsupervised learning allows models to learn better initial parameters from unlabeled datasets, fine-tuning with labeled data is still required to 'guide' the model towards its learning goal.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The labeling process for video-based affective datasets, especially with continuous annotations, is expensive and cannot easily be automated.

“The labeling process for video-based affective datasets, especially with continuous annotations, is expensive and cannot easily be automated.”

Probably not a substantial claim: may not pass S2 (checkable).

Such results indicate that the study of DNNs in affective computing could contribute to interdisciplinary emotion research by giving a further perspective on affective displays and their representation in general.

“Such results indicate that the study of DNNs in affective computing could contribute to interdisciplinary emotion research by giving a further perspective on affective displays and their representation in general.”

May combine two claims.

Unsupervised learning has proved successful in SER; Deng et al. improved the training process on labeled data by exploiting knowledge from unlabeled data using AEs.

“Unsupervised learning has also proved successful in SER-for example, Deng et al. [140] were able to improve the training process on labeled data by exploiting knowledge from unlabeled data using AEs.”

• cites [140] [140] J. Deng, X. Xu, Z. Zhang, S. Fr¨ uhholz, and B. Schuller, 'Universum autoencoder-based domain adaptation for speech emotion recognition,' IEEE Signal Process. Lett. , vol. 24, no. 4, pp. 500-504, 2017. -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

There are few specialized architectures researchers can draw from, which is why many fall back to ones from object recognition.

“There are few specialized architectures researchers can draw from, which is why many fall back to ones from object recognition.”