CoChat
Sign Up
Cited by 86Emotion and Mood RecognitionSocial Robot Interaction and HRIFace and Expression Recognition

Multimodal emotional state recognition using sequence-dependent deep hierarchical features

Open access
Barros, Pablo, Jirak, Doreen, Weber, Cornelius, Wermter, Stefan·2015·Neural Networks
DOI 10.1016/j.neunet.2015.09.009
Open this paper in CoChat — free accountOpen PDF

Abstract

Emotional state recognition has become an important topic for human-robot interaction in the past years. By determining emotion expressions, robots can identify important variables of human behavior and use these to communicate in a more human-like fashion and thereby extend the interaction possibilities. Human emotions are multimodal and spontaneous, which makes them hard to be recognized by robots. Each modality has its own restrictions and constraints which, together with the non-structured behavior of spontaneous expressions, create several difficulties for the approaches present in the literature, which are based on several explicit feature extraction techniques and manual modality fusion. Our model uses a hierarchical feature representation to deal with spontaneous emotions, and learns how to integrate multiple modalities for non-verbal emotion recognition, making it suitable to be used in an HRI scenario. Our experiments show that a significant improvement of recognition accuracy is achieved when we use hierarchical features and multimodal information, and our model improves the accuracy of state-of-the-art approaches from 82.5% reported in the literature to 91.3% for a benchmark dataset on spontaneous emotion expressions.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes a Multichannel Convolutional Neural Network (MCCNN) for spontaneous multimodal emotion recognition from face expression and body motion, using hierarchical features and learned modality fusion. On the FABO benchmark, the multimodal model reaches 91.30% accuracy, improving on the 82.5% state-of-the-art result reported in the literature.

Key findings

  • Multimodal recognition performed best: 91.30% (±2.7) accuracy, compared with 72.70% (±3.1) for facial expression alone and 57.84% (±7.7) for body motion alone.
  • Parameter exploration found the best result, 91.30%, with 10 filter maps and 3×3 receptive fields in both layers; larger receptive fields tended to reduce accuracy, with the worst reported configuration at 87.89%.
  • The MCCNN improved over state-of-the-art multimodal results: 91.3% versus 82.65% for Gunes and Piccardi (2009) and more than 75% for Chen et al. (2013) using bag-of-words.
  • Common CNN channels were weaker than the MCCNN in the multimodal experiment: gray scale 74.5%, Sobel X 69.7%, and Sobel Y 71.2%.

Methods

  • The model is an MCCNN with three channels: raw gray-scale images, Sobel X, and Sobel Y; a cubic receptive field is used in the first layer for sequence dependency, and the channels are fused through a fully connected layer feeding a logistic regression classifier.
  • Experiments use the FABO bimodal face and body benchmark: 281 upper-body videos with majority temporal-phase voting, 10 emotional states plus a grouped “Neutral” class, and only apex frames used for training; three-fold cross validation is performed and averages over 30 trials are reported.
  • Three experiments compare face expression, body motion, and multimodal input; face is tracked with Viola-Jones, body motion is represented by summed absolute frame differences with weighted shadows, and multimodal input uses raw upper-body frames.

Limitations

  • Stated: body motion alone gave the worst result, and the authors say the model needs a more structured motion representation for gesture recognition; they also note future work is needed to visualize learned features and test real-world real-time recognition.
  • Stated: parameter evaluation was done on FABO, where subjects vary in movement, clothing, accessories, gender, age, and ethnicity; adapting topology and tuning could address distance, multiple subjects, and occlusion.
  • Evident from the text: only apex frames are used for training while other temporal phases are grouped into “Neutral,” and the Neutral class is the most misclassified because it is the largest and contains examples from every subject; body motion also uses a single motion image without the cubic receptive field, limiting temporal modeling.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.neunet.2015.09.009

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 54 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The proposed model improves the accuracy of state-of-the-art approaches from 82.5% reported in the literature to 91.3% for a benchmark dataset on spontaneous emotion expressions.

    “Our experiments show that a significant improvement of recognition accuracy is achieved when we use hierarchical features and multimodal information, and our model improves the accuracy of state-of-theart approaches from 82.5% reported in the literature to 91.3% for a benchmark dataset on spontaneous emotion expressions.”
    Quote in the paperSubstantial claim
  • Findingown data

    The experiments show that a significant improvement of recognition accuracy is achieved when using hierarchical features and multimodal information.

    “Our experiments show that a significant improvement of recognition accuracy is achieved when we use hierarchical features and multimodal information, and our model improves the accuracy of state-of-theart approaches from 82.5% reported in the literature to 91.3% for a benchmark dataset on spontaneous emotion expressions.”
    Quote in the paperSubstantial claim
  • Findingown data

    The results show that the proposed approach is more appropriate for spontaneous emotion recognition than the models using manual feature fusion.

    “Our results show that our approach is more appropriate for spontaneous emotion recognition than the models using manual feature fusion.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    After training, the third channel will have developed a more complex edge representation than Gabor filters.

    “After training, the third channel will have developed a more complex edge representation than Gabor filters.”
    Quote in the paperNot a substantial claim
  • Findingreasoning

    The three channels influence each other, driving the filters' training to a different direction when only one channel is trained.

    “The three channels influence each other, driving the filters' training to a different direction when only one channel is trained.”
    Quote in the paperSubstantial claim
  • May combine two claims. 67
  • Classified as probably not a substantial claim. 44

  • May combine two claims.

    Findingown data

    The channels 2 and 3 produced edge-detectors with a similar structure to the Sobel filters, and this structure emerged during training.

    “We can see that the channels 2 and 3 produced edge-detectors with a similar structure of the Sobel filters, and this structure emerged during training.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Channel 1 produced more complex filters, with structures similar to both channels combined.

    “Channel 1 produced more complex filters, with structures similar to both channels combined.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Some of the sequences contain head motion and are occluded by hands.

    “Some of the sequences also contain head motion and are occluded by hands.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    The apex phase of each expression can vary from six to fifteen frames.

    “The apex phase of each expression can vary from six to fifteen frames.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    When using 10 filter maps, an average of 4327.88 s was necessary for training, which was faster than using 20 (7253.74 s) and 30 (12 873.64 s).

    “When using 10 filter maps an average of 4327.88 s (1 h 12 min) was necessary for training which was faster than using 20, with an average of 7253.74 s (2 h), and 30 filter maps with an average of 12 873.64s (3 h 30 min).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When using smaller receptive fields, the network produces results with a better accuracy median and smaller variance.

    “When using smaller receptive fields, the network produces results with a better accuracy median and smaller variance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    After the first layer it is difficult to interpret the representation, but as each layer receives the representation from a previous layer, the network encodes hierarchical representation.

    “After the first layer it is difficult to interpret the representation, but as each layer receives the representation from a previous layer, the network encodes hierarchical representation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Our model achieves higher accuracy for the experiments with Face expression and Multimodal experiments, compared with the results reported in Chen et al. (2013).

    “Our model achieves higher accuracy for the experiments with Face expression and Multimodal experiments, compared with the results reported in Chen et al. (2013).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Chen et al. (2013) source

      Not checked in the free run.

  • May combine two claims.

    Findingnone given

    When increasing the number of filter maps, the average of the results is also higher, despite the best accuracy being lower.

    “When increasing the number of filter maps, the average of the results is also higher, despite the best accuracy being lower.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    The recognition time for all three experiments was very similar, around 6 ms, due to using the same network topology and input image size.

    “The recognition time for all three experiments was very similar, around 6 ms, due to using the same network topology and input image size.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The minimum and maximum values of 10 and 30 filter maps were chosen based on preliminary experiments, where these values represented the limits where the network showed a big variation for the accuracy.

    “The minimum and maximum values of 10 and 30 filter maps were chosen based on preliminary experiments, where these values represented the limits where the network showed a big variation for the accuracy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    The training time was similar for the Facial Expression and Multimodal experiments, because both experiments used the same number of images.

    “The training time was similar for the Facial Expression and Multimodal experiments, because both experiments used the same number of images.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Although the accuracy is similar when using any of the three filter map numbers, the training time when using 10 filter maps is smaller.

    “Although the accuracy is similar when using any of the three filter map numbers, the training time when using 10 filter maps is smaller.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the Multimodal experiment, MCCNN increased accuracy by almost 10% compared to Gunes and Piccardi (2009) and more than 15% compared to Chen et al. (2013).

    “Our strategy presented best results, and increased the accuracy by almost 10 % compared to Gunes and Piccardi (2009) and more than 15 % in comparison with Chen et al. (2013).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Gunes and Piccardi (2009)

      Not checked in the free run.

    • cites Chen et al. (2013) source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    In the Facial Expression experiment, MCCNN achieved higher accuracy than Chen et al. (2013) and Gunes and Piccardi (2009).

    “Our model was measured with a higher accuracy when compared with the approaches of Chen et al. (2013) and Gunes and Piccardi (2009), showing that our feature learning is more reliable.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Chen et al. (2013) source

      Not checked in the free run.

    • cites Gunes and Piccardi (2009)

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    The common CNN architecture achieved a lower accuracy of 53.32% when using the gray scale image, and 51.35% and 51.85% when using the Sobel in X and Y directions respectively.

    “The common CNN architecture also achieved a lower accuracy of 53.32% when using the gray scale image, and 51.35% and 51.85% when using the Sobel in X and Y directions respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For the body motion experiment, a common CNN is enough to learn information from a motion image, and the application of the Sobel filter is not necessary for this case.

    “For the body motion experiment, we showed that a common CNN is enough to learn information from a motion image, and the application of the Sobel filter is not necessary for this case.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The first layers extract complex shapes like eyes, mouths and eyebrows, which are used by deeper layers to build more complex representations.

    “The first layers are able to extract complex shapes, like eyes, mouths and eyebrows, which are used for the deeper layers to build more complex representation.”
    • Also stated in: § 5. Conclusion and future works · plain
      “The first channel extracts more complex structures than the other two, as regions with eyes or mouth.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The model can recognize a sequence of emotions from different subjects, expressed spontaneously, without any kind of previously determined structure, and in different positions from the camera.

    “The model can recognize a sequence of emotions from different subjects, expressed spontaneously, without any kind of previously determined structure, and in different positions from the camera.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The FABO dataset's emotional expressions are expressed spontaneously and without any predetermined structure.

  • May combine two claims.

    Findingown data

    Our model achieves a total of 91.3% accuracy, almost 10% more than the one reported by Gunes and Piccardi (2009), without using any explicit feature selection.

    “Our model achieves a total of 91.3% accuracy, almost 10% more than the one reported by Gunes and Piccardi (2009), without using any explicit feature selection.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gunes and Piccardi (2009)

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Each layer is capable of extracting relevant information from a non-structured emotion expression, being able to identify which characteristics of the input stimuli are the most important.

    “Each layer is capable to extract relevant information from a non-structured emotion expression, being able to identify which characteristics of the input stimuli is the most important.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The network's hierarchical topology is kept the same during all three experiments, but the input stimuli change; this provides a robust evaluation scheme since the only aspect changing is the presented stimuli.

    “The network's hierarchical topology is kept the same during all three experiments, but the input stimuli change.This shows the capacity of the model to learn from different inputs, and provides a robust evaluation scheme since the only aspect changing is the presented stimuli.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The Sobel filters help the model create a more complex and specific representation of the visual stimuli, accelerating the learning process.

    “The Sobel filters present this rudimentary information, and help the model creating a more complex and specific representation of the visual stimuli, accelerating the learning process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Deeper layers learn how to extract detailed structures like eyes, mouth, hand movements and so on.

    “Deeper layers learn how to extract detailed structures like eyes, mouth, hand movements and so on.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The use of supervised training allows training the network with a smaller amount of data, which would not be possible when using unsupervised training.

    “The use of supervised training allows us to train the network with a smaller amount of data, which would not be possible when using unsupervised training.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    By learning which characteristic is more relevant, the proposed model passes them to deeper layers, and thus realizes a learned modality fusion.

    “By learning which characteristic is more relevant, our model passes them to deeper layers, and thus realizes a learned modality fusion.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The first layers learn how to extract macro regions which contain important information, for example ignoring the background.

    “The first layers learn how to extract macro regions which contain important information, for example ignoring the background.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The proposed model does not rely on several different feature extraction techniques, but lets the model learn in each layer which are the most relevant features.

    “This way, we do not rely on several different feature extraction techniques, but let the model learn in each layer which are the most relevant features.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The multichannel implementation improves the capacity of the network to adapt to each stimulus, and learn hierarchical features that are unique for each modality.

    “This capability improves the capacity of the network to adapt to each stimulus, and learn hierarchical features that are unique for each modality.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The shared representation is learned by the network, and consists of a new representation of both modalities.

    “Our shared representation is learned by the network, and consists of a new representation of both modalities.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The proposed model is able to learn simple and complex features and to model the dependencies of these features in a sequence.

    “Our proposed model is able to learn simple and complex features and to model the dependencies of these features in a sequence.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The proposed system is capable of generating a specific edge-like representation for each stimulus in the first layers and a complex representation of emotional states in the deeper ones.

    “The proposed system is capable to generate a specific edge-like representation for each stimulus in the first layers and a complex representation of emotional states in the deeper ones.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Different from previous approaches, the proposed system can create a shared representation of both modalities, without relying on manual feature fusion techniques.

    “Different from previous approaches, our proposed system can create a shared representation of both modalities, without relying on manual feature fusion techniques.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The proposed model learns how to fuse each multimodal stream by using a fully connected layer, removing the constraints of each stream and generating a set of robust features.

    “Our model learns how to fuse each multimodal stream by using a fully connected layer, removing the constraints of each stream and generating a set of robust features.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The representation developed by the third channel is very specific for the kind of stimulus presented to the network, which improves the final representation obtained by the model even when using fewer layers.

    “This representation turns out to be very specific for the kind of stimulus that is presented to the network, which improves the final representation obtained by the model even when using fewer layers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The FABO database is composed of recordings of the face and body motion using two cameras, one capturing only the face and the second capturing the upper body.

    “The database is composed of recordings of the face and body motion using two cameras, one of them capturing only the face and the second one capturing the upper body.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Gunes and Piccardi (2006)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The Viola-Jones algorithm is robust and effective when applied in general face detection datasets and even in real-world scenarios.

    “Wang (2014) discuss the Viola-Jones algorithm, and shows that it is robust and effective when applied in general face detection datasets, and even in real-world scenarios.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wang (2014) source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The brain is capable of integrating multimodal information and generating a representation for visual stimuli based on all of them together.

    “The brain is also capable to integrate this multimodal information and generate a representation for the visual stimuli based on all of them together.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Adolphs (2002) source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    When none of the emotion expressions is recognized, the model generates the Neutral label, allowing temporal classification of emotions in an online classification scenario.

    “When none of the emotion expressions is recognized, our model will generate the ''Neutral'' label, which allows performing a temporal classification of emotions in an online classification scenario.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The multimodal experiment does not perform manual feature fusion, but focuses on the learning process and the use of both face and body information to create a unique representation.

    “Different from other approaches, this experiment does not perform manual feature fusion, but focuses on the learning process and the use of both, face and body information to create a unique representation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Fasel's CNN model for face expression recognition can only extract information from separated parts of the face at a time and sum them at the end, not taking into consideration the interactions and relations between each observed characteristic.

    “However, their model can only extract information from separated parts of the face at a time and sum them at the end to represent the facial expression.This process does not take into consideration the interactions and relations between each observed characteristic.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Fasel (2002)

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Using the sequence dependency approach, the model can be fed with a live stream of frames and identify when an emotion is demonstrated, discerning different emotions by their apex phases.

    “Using this approach, we can feed our model with a live stream of frames, and it will be able to identify when an emotion is demonstrated, discerning different emotions by their apex phases.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: In the apex phase, the facial expression and motion of the subject are more intense and display unique body postures and face expressions within each emotion.
    • Rests on an assumption: The temporal features contained in a sequence of frames during the temporal phases are important and must be present in the sequence representation.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    The absolute difference of each pair of frames removes non-changing parts of the image, being able to remove the background or any other detail not important for the motion representation.

    “The absolute difference of each pair of frames removes non-changing parts of image, being able to remove the background or any other detail in the image that is not important for the motion representation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Convolutional Neural Networks are inspired by the hierarchical process of simple and complex cells in the human brain to extract and learn different information from visual stimuli.

    “Convolutional Neural Networks (CNN) were introduced formally by Lecun, Bottou, Bengio, and Haffner (1998).They are inspired by the hierarchical process of simple and complex cells in the human brain to extract and learn different information from visual stimuli.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lecun, Bottou, Bengio, and Haffner (1998) source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    After training, the model can recognize occluded, partial and rotated faces present in the evaluated dataset.

    “so that after training, the model can recognize occluded, partial and rotated faces which are present in the evaluated dataset.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    In non-verbal communication, facial expressions and body posture/motion complement each other when determining emotional states.

    “They discuss that in non-verbal communication, facial expressions and body posture/motion complement each other when determining emotional states.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gu, Mai, and Luo (2013)

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    In an emotion sequence, there are four temporal phases: onset, apex, offset, and neutral.

    “In an emotion sequence, there are four temporal phases: onset, apex, offset, and neutral, as illustrated in Fig. 4 and discussed by Gunes and Piccardi (2009)”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gunes and Piccardi (2009)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    In the experiments, the Viola-Jones algorithm was 100% correct and detected the positions and sizes of the faces in all presented images.

    “In our experiments, the Viola-Jones algorithm was 100% correct and detected the positions and sizes of the faces in all presented images.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The motion representation contains the shape of the motion and, with the help of the weighted shadows, the information of when each single posture happened.

    “This representation contains the shape of the motion and, with the help of the weighted shadows, the information of when each single posture happened.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Each video contains one subject executing the same expression in a cycle of two to four expressions per video.

    “Each video contains one subject executing the same expression in a cycle of two to four expressions per video.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Gunes and Piccardi (2006)

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Facial expressions are included in most emotional demonstrations and present the strongest impact when inferring the emotional state of a human.

    “Facial expressions are included in most of the emotional demonstrations, and are the characteristics which present the strongest impact when inferring the emotional state of a human.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Cabanac (2002) source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Each person demonstrates his/her own emotions using hands and arms and movements, which contain no structure and are highly dependent on the person's personality and attitude towards the situation.

    “Each person demonstrates his/her own emotions using hands and arms and movements, which contain no structure and are highly dependent on the person's personality and attitude towards the situation, as discussed by Zhao and Badler (1998).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Zhao and Badler (1998)

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    When presented together, facial expressions and body posture/motion are recognized differently than when they are shown individually, which shows a dependency within the modalities that changes emotional recognition.

    “They show that when presented together, both modalities are recognized differently when they are shown individually, which shows a dependency within the modalities that changes the emotional recognition.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gu, Mai, and Luo (2013)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Because the network is not very deep, increasing the number of filter maps is needed to expand the variety of features extracted in the same layer.

    “But, as our network is not very deep, we need to increase the number of filter maps in order to expand the variety of features extracted in the same layer.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The three parameters were chosen because of their major influence on the network response.

    “The three parameters were chosen because of their major influence on the network response.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    A total of 100 training epochs were performed for each experiment, which was shown to be enough to achieve convergence.

    “A total of 100 training epochs were performed for each experiment, which was shown to be enough to achieve convergence.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    The number of filter maps affects directly the amount of features extracted, and what these features represent.

    “The number of filter maps affects directly the amount of features extracted, and what these features represent.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    A large number of feature maps introduces redundancy, and a small number is not enough to extract a proper description of the emotion sequence.

    “A large number of feature maps introduces redundancy, and a small number is not enough to extract a proper description of the emotion sequence.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Because of the representation of motion in one single frame and the use of a common receptive field instead of a cubic one, the motion experiment has less input stimuli and less parameters to train.

    “Because of the representation of motion in one single frame and the use of a common receptive field instead of a cubic one, the motion experiment has less input stimuli and less parameters to train.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Extending the number of filter maps causes the network to generate different feature representation at the same level and thus generate redundant features which allow better generalization.

    “This occurs because when extending the number of filter maps, the network generates different feature representation at the same level and thus generates redundant features which allow a better generalization.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    In the Facial Expression experiment, only the face was present, which leads to faster convergence because the variations within each class rely on the face expression itself, and not on different subjects, clothes or any other part of the image.

    “In the Facial Expression experiment only the face was present, which leads to faster convergence because the variations within each class rely on the face expression itself, and not on different subjects, clothes or any other part of the image.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The size of the receptive fields determines which pixel structures are important for the model.

    “The size of the receptive fields determines which pixel structures are important for the model.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The CNN has competitive results compared to the models of Chen et al. (2013) and Gunes and Piccardi (2009), and the MCCNN implementation outperformed these models.

    “This shows that the CNN has competitive results, when compared to the models of Chen et al. (2013) and Gunes and Piccardi (2009), and when using the MCCNN implementation, it was possible to outperform these models.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Chen et al. (2013) source

      Not checked in the free run.

    • cites Gunes and Piccardi (2009)

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Although the results of our model are not as good as in these experiments, there are no pre-processing steps on the images and thus less computational processing is necessary.

    “Although the results of our model are not as good as in these experiments, there are no pre-processing steps on the images and thus less computational processing is necessary.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    If the receptive fields are too large, they will absorb more pixel structures than necessary, and they will not be able to determine or react to these structures, aggregating more than one structure into one filter map.

    “If they are too large, they will absorb more pixel structures than necessary, and they will not be able to determine or to react to these structures, aggregating more than one structure into one filter map.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    For all experiments, three-fold cross validation is performed, and the average performance over 30 trials is reported.

    “For all experiments, three-fold cross validation is performed, and the average performance over 30 trials is reported in this work.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    If the receptive fields are too small, they will not be able to enhance important pixel structures, and will generate redundancy for the next layers.

    “If the receptive fields are too small, they will not be able to enhance important pixel structures, and will generate redundancy for the next layers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Passing general features through the layers generates higher level features in deeper layers.

    “Passing these general features through our layers generates higher level features in deeper layers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    On the first layer, the receptive fields are connected directly to the image, and they will enhance structures present in the original data.

    “On the first layer, the receptive fields are connected directly to the image, and they will enhance structures present in the original data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    As no structured motion is present, the body movements act like a complement for facial expression.

    “As no structured motion is present, the body movements act like a complement for facial expression.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    With too many redundant filters, the network loses the capability of generalizing and ends up overfitting.

    “It is important to note that with too many redundant filters, the network lose the capability of generalizing and ends up overfitting.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    All the channels, individually, obtained a similar accuracy, indicating that the application of Sobel filters in the image does not improve the recognition of motion images.

    “Also, all the channels, individually, obtained a similar accuracy, indicating that the application of Sobel filters in the image does not improves the recognition of motion images.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The third channel specializes on more complex features because the other two channels receive Sobel-filtered images.

    “As the two channels receive the images from the Sobel filter layers, the remaining one will be influenced, and will specialize on more complex features.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Increasing the number of filter maps increases connections and parameters to be updated during training, which could lead to overfitting.

    “The drawback of increasing the number of filter maps is an increase of the connections and the parameters to be updated during training, which could lead to overfitting.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The network was evaluated with an established dataset in three different experiments.

    “The network is evaluated with an established dataset in three different experiments.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Chen et al. (2013) use 53 face landmarks to extract features and classify them with an SVM.

    “In the research of Chen et al. (2013), they use 53 face landmarks to extract some features and classify them with an SVM.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Chen et al. (2013) source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    This process reduces the computational power necessary for training and recognition, and eliminates constraints of using specific features such as illumination change, different skin color tones and occlusion.

    “This process reduces the computational power necessary for training and recognition, and eliminates the constraints of using specific features, which could be listed in this experiment as illumination change, different skin color tones and occlusion among others.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    For the upper-body images, the face is not the most prominent region in the image and when the Sobel filters are applied, the image loses some of the small structures that compose the face.

    “This is explained by the fact that for the upper-body images the face is not the most prominent region in the image and when the Sobel filters are applied, the image loses some of the small structures that composes the face.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    Our model was evaluated using the same methodology for experiments established by Gunes and Piccardi (2009).

    “Our model was evaluated using the same methodology for experiments established by Gunes and Piccardi (2009).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gunes and Piccardi (2009)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The worse Body Motion results may be because the proposed model needs a more structured motion representation to be used in a gesture recognition task.

    “This could be due to the fact that to be used in a gesture recognition task, the proposed model needs a more structured motion representation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The Neutral class is the most misclassified because it has the largest number of sequences and contains examples of every subject from all temporal phases except the apex phase.

    “This class has the largest number of sequences and contains examples of every subject in the dataset, from all temporal phases except the apex phase, which explains why it is the most misclassified class.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The misclassifications between Boredom, Anxiety, Puzzlement and Uncertainty in the Facial Expression experiment occur because these classes have similar face expressions.

    “This happens, because these classes have similar face expressions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Gunes and Piccardi (2009) create a motion representation by tracking facial landmarks, shoulder and hands, applying 144 different feature extraction techniques to each body part.

    “The work of Gunes and Piccardi (2009) creates a motion representation tracking different parts of the human: facial land marks, shoulder and hands.Their representation is obtained by the application of 144 different feature extraction techniques in each body part.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Gunes and Piccardi (2009)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Increasing the number of filter maps increases the number of features extracted from the image.

    “Increasing the number of filter maps will generate more different filters, which will increase the number of features extracted from the image.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    The representations of emotions learned by the proposed model are independent of specific feature extraction techniques, and independent of background and foreground segmentation.

    “Also, the representations of the emotions learned by the proposed model are independent of specific feature extraction techniques, and independent of background and foreground segmentation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Our model does not use specific predefined features; the images are presented to the network, and the model learns the most relevant features for emotional state classification.

    “Our model does not use specific predefined features.The images are presented to the network, and the model learns which are the most relevant features in the image for the emotional state classification task.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Chen et al. (2013) apply skin-color segmentation, head and hand tracking, and generate motion history images to represent body movement.

    “In the work of Chen et al. (2013) they apply skin-color segmentation, head and hand tracking techniques to create a segmented image.After that, they generate motion history images to represent the body movement.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Chen et al. (2013) source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The higher accuracy of MCCNN in the Facial Expression experiment shows that its feature learning is more reliable.

    “Our model was measured with a higher accuracy when compared with the approaches of Chen et al. (2013) and Gunes and Piccardi (2009), showing that our feature learning is more reliable.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Gunes and Piccardi (2009) extract a series of features from the face by tracking regions and applying 150 different feature descriptors per region, which consumes a lot of computational power for feature extraction.

    “To evaluate the Face expression,Gunes2009 extract a series of features from the face.They first track eyes, eyebrows, mouth, nose, chin, and other face regions.For each region, they apply a series of 150 different feature descriptors.This process consumes a lot of computational power for feature extraction.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gunes and Piccardi (2009)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    This result agrees with psychological and social studies of Gu et al. (2013) and Kret et al. (2013).

    “This result agrees with psychological and social studies of Gu et al. (2013) and Kret et al. (2013).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Gu et al. (2013)

      Not checked in the free run.

    • cites Kret et al. (2013) source

      Not checked in the free run.

90 of 90

  • The paper relies on this without support.

    Human emotions are multimodal and spontaneous, which makes them hard to be recognized by robots.

    “Human emotions are multimodal and spontaneous, which makes them hard to be recognized by robots.”

  • The paper relies on this without support.

    Each modality has its own restrictions and constraints which, together with the non-structured behavior of spontaneous expressions, create several difficulties for the approaches present in the literature, which are based on several explicit feature extraction techniques and manual modality fusion.

    “Each modality has its own restrictions and constraints which, together with the non-structured behavior of spontaneous expressions, create several difficulties for the approaches present in the literature, which are based on several explicit feature extraction techniques and manual modality fusion.”

  • The paper relies on this without support.

    In the apex phase, the facial expression and motion of the subject are more intense and display unique body postures and face expressions within each emotion.

    “In this phase, the facial expression and motion of the subject are more intense and display unique body postures and face expressions within each emotion.”

  • The paper relies on this without support.

    The temporal features contained in a sequence of frames during the temporal phases are important and must be present in the sequence representation.

    “The temporal features contained in a sequence of frames during the temporal phases are important and must be present in the sequence representation.”

  • The paper relies on this without support.

    Manual feature fusion requires well-defined feature representations for both body motion and facial expression to show good results.

    “This manual feature fusion approach showed good results, but only if the feature representation for both body motion and facial expression were well defined.”

  • The paper relies on this without support.

    Increasing the number of features to be classified can lead to redundant or concurrent features.

    “The increase in the number of features to be classified has to be taken into consideration, which could lead to redundant or concurrent features.”

  • The paper relies on this without support.

    The FABO dataset's emotional expressions are expressed spontaneously and without any predetermined structure.

    “The model can recognize a sequence of emotions from different subjects, expressed spontaneously, without any kind of previously determined structure, and in different positions from the camera.”
7 of 7

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Kernel sizes assigned to the 87.89% result are reversed

    Minor

    The text assigns 87.89% to an 11 × 11 first-layer and 21 × 21 second-layer configuration. Table 3 assigns that result to first-layer size 21 and second-layer size 11; Table 2 does not list 21 × 21 as a second-layer option.

  • Reported worst result for ten filter maps is not the lowest tabulated result

    Minor

    The text calls 87.89% the worst result with ten first-layer filter maps, but Table 3 also reports 87.34% for ten filter maps.

  • Prior best accuracy is given as both 82.5% and 82.65%

    Minor

    The abstract gives 82.5% as the prior reported accuracy, while the discussion gives 82.65% as the best accuracy from Gunes and Piccardi (2009). These do not agree even when 82.65% is rounded to one decimal place.

3 of 3

Cites

Flagged first, then by citations

2 of the 46 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Barros, P., Jirak, D., Weber, C., & Wermter, S.
Affiliations
Universität Hamburg (DE)
Corresponding author
Barros, Pablo
ORCID
Venue
Neural Networks
Publisher
Elsevier BV
Published
2015-12
Volume
72
Pages
140-151
ISSN
0893-6080 (ISSN-L)
DOI
10.1016/j.neunet.2015.09.009
OpenAlex
W1871419576
Semantic Scholar
ac3682436e78bdce3905564d1cacdfbad785d0c2
Cited by
86
References
46
Citations by year
2026: 4 · 2025: 7 · 2024: 4 · 2023: 7 · 2022: 2 · 2021: 18 · 2020: 11 · 2019: 12 · 2018: 11 · 2017: 6 · 2016: 3 · 2015: 1
FWCI
10.54
Type
journal article
Language
en
Open access
Licence
Locations
  • Neural Networks (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
Funders
CAPES Brazilian Federal Agency (5951–13–5)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
emotion recognition, multimodal emotion recognition, human-robot interaction, hierarchical feature representation, modality fusion
MeSH terms
Emotions, Humans, Learning, Robotics, Recognition, Psychology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review