CoChat

Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition

Ghaleb, Esam, Popa, Mirela, Asteriadis, Stylianos · 2019 · 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)
Cited by 93

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Abstract

In Audio-Video Emotion Recognition (AVER), the idea is to have a human-level understanding of emotions from video clips. There is a need to bring these two modalities into a unified framework, to effectively learn multimodal fusion for AVER. In addition, literature studies lack in-depth analysis and utilization of how emotions vary as a function of time. Psychological and neurological studies show that negative and positive emotions are not recognized at the same speed. In this paper, we propose a novel multimodal temporal deep network framework that embeds video clips using their audio-visual content, onto a metric space, where their gap is reduced and their complementary and supplementary information is explored. We address two research questions, (1) how audio-visual cues contribute to emotion recognition and (2) how temporal information impacts the recognition rate and speed of emotions. The proposed method is evaluated on two datasets, CREMA-D and RAVDESS. The study findings are promising, achieving the state-of-the-art performance on both datasets, and showing a significant impact of multimodal and temporal emotion perception.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/acii.2019.8925444

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 7 references

Research works printed but not linked: 25

  1. beard, 0, Multi-modal sequence fusion via recursive attention for emotion recognition, Proceedings of the Conference on Computational Natural Language Learning, 251
  2. aytar, 2016, Soundnet: Learning sound representations from unlabeled video, Advances in neural information processing systems, 892
  3. 10.1016/S0167-6393(02)00077-8 · DOI
  4. 10.1145/3240508.3240578 · DOI · Paper page
  5. 10.1145/2993148.2993151 · DOI
  6. 10.1145/2993148.2997632 · DOI · Paper page
  7. 10.1080/016909696386999 · DOI
  8. weinberger, 2009, Distance metric learning for large margin nearest neighbor classification, Journal of Machine Learning Research, 10, 207
  9. xu, 2012, Distance metric learning for kernel machines
  10. 10.1109/MSP.2017.2732900 · DOI
  11. 10.1371/journal.pone.0196391 · DOI · Paper page
  12. 10.1109/TAFFC.2014.2336244 · DOI · Paper page
  13. zheng, 2017, Identifying stable patterns over time for emotion recognition from eeg, IEEE Transactions on Affective Computing
  14. 10.1111/bjet.12324 · DOI
  15. 10.1145/3219819.3219856 · DOI
  16. 10.1109/IROS.2018.8593503 · DOI
  17. lee, 2017, Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers, IEEE Transactions on Human-Machine Systems
  18. 10.1177/0023830909348993 · DOI
  19. 10.1016/j.comcom.2016.03.010 · DOI · Paper page
  20. 10.1109/TAFFC.2018.2890471 · DOI · Paper page
  21. ngiam, 0, Multimodal deep learning, Proceedings of the 28th International Conference on Machine Learning (ICML-11), 689
  22. li, 2018, A survey of multi-view representation learning, IEEE Transactions on Knowledge and Data Engineering
  23. 10.1109/JSTSP.2018.2875385 · DOI · Paper page
  24. 10.1109/TCYB.2016.2591583 · DOI · Paper page
  25. sohn, 2016, Improved deep metric learning with multi-class n-pair loss objective, Advances in neural information processing systems, 1857

Cited by

See all 93

Metadata

Authors
Ghaleb, E., Popa, M., & Asteriadis, S.
Venue
2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)
Publisher
IEEE
Published
2019-09
Pages
552-558
OpenAlex
W2977259558
Cited by
93
References
7
FWCI
17.14
Type
conference paper
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition. https://app.cochat.ai/papers/10.1109/acii.2019.8925444
The paper
Ghaleb, E., Popa, M., & Asteriadis, S. (2019). Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition. 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). https://doi.org/10.1109/acii.2019.8925444

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.