CoChat

Multimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals

Open access

The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.

Abstract

Emotion recognition is a very important technique for ultimate interactions between human beings and artificial intelligence systems. For effective emotion recognition in a continuous-time domain, this article presents a multimodal fusion network which integrates video modality and electroencephalogram (EEG) modality networks. To calculate the attention weights of facial video features and the corresponding EEG features in fusion, a multimodal attention network, that is utilizing bilinear pooling based on low-rank decomposition, is proposed. Finally, continuous domain valence values are computed by using two modality network outputs and attention weights. Experimental results show that the proposed fusion network provides an improved performance of about 6.9% over the video modality network for the MAHNOB human computer interface (MAHNOB-HCI) dataset. Also, we achieved the performance improvement even for our proprietary dataset.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/access.2020.3036877

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Figures

FIGURE 1 A general fully connected layer based multimodal deep learning structure.
FIGURE 1 A general fully connected layer based multimodal deep learning structure.CC-BY · From the PDF
FIGURE 2 A block diagram of the proposed method.
FIGURE 2 A block diagram of the proposed method.CC-BY · From the PDF
FIGURE 3 A block diagram of the video modality network.
FIGURE 3 A block diagram of the video modality network.CC-BY · From the PDF
FIGURE 4 The EEG feature extraction process and the LSTM model of EEG modality network.
FIGURE 4 The EEG feature extraction process and the LSTM model of EEG modality network.CC-BY · From the PDF
FIGURE 5 The EEG feature extraction process and the LSTM model of EEG modality network.
FIGURE 5 The EEG feature extraction process and the LSTM model of EEG modality network.CC-BY · From the PDF
FIGURE 6 Histogram of valence label. (a) MAHNOB-HCI dataset, (b) ASIA dataset.
FIGURE 6 Histogram of valence label. (a) MAHNOB-HCI dataset, (b) ASIA dataset.CC-BY · From the PDF
FIGURE 7 Experimental protocol of a trial with the ASIA dataset.
FIGURE 7 Experimental protocol of a trial with the ASIA dataset.CC-BY · From the PDF
FIGURE 8 Valance result plotting of the MAHNOB-HCI dataset: (a) sequence index: 01_08, (b) sequence index: 07_34, and (c) sequence index: 24_22.
FIGURE 8 Valance result plotting of the MAHNOB-HCI dataset: (a) sequence index: 01_08, (b) sequence index: 07_34, and (c) sequence index: 24_22.CC-BY · From the PDF
FIGURE 9 RMSE vs. valence of video and EEG modality. For this experiment, MAHNOB-HCI dataset was used.
FIGURE 9 RMSE vs. valence of video and EEG modality. For this experiment, MAHNOB-HCI dataset was used.CC-BY · From the PDF
FIGURE 11 Valance results for the ASIA dataset: (a) an example of a positive reaction, and (b) an example of a negative reaction.
FIGURE 11 Valance results for the ASIA dataset: (a) an example of a positive reaction, and (b) an example of a negative reaction.CC-BY · From the PDF
FIGURE 10 (a) Valence output for sample 23_20 of the MAHNOB dataset, and (b) fusion weights of the two modalities.
FIGURE 10 (a) Valence output for sample 23_20 of the MAHNOB dataset, and (b) fusion weights of the two modalities.CC-BY · From the PDF
TABLE 5. Performance comparison according to modality and fusion style for the ASIA dataset.
TABLE 5. Performance comparison according to modality and fusion style for the ASIA dataset.CC-BY · From the PDF
Image 13
CC-BY · From the PDF
Image 14
CC-BY · From the PDF
Image 15
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 3

  1. zhang, 2017, S3FD: Single shot scale-invariant face detector, Proc IEEE Int Conf Comput Vis, 192
  2. valstar, 2010, Induced disgust, happiness and surprise: An addition to the MMI facial expression database, Proc Third Int'l Workshop EMOTION (satellite of LREC) Corpora for Research on Emotion and Affect, 65
  3. cukic, 2018, EEG machine learning with Higuchi fractal dimension and sample entropy as features for successful detection of depression, arXiv 1803 05985

Cited by

See all 39

Metadata

Authors
Choi, D. Y., Kim, D.-H., & Song, B. C.
Affiliations
Inha University (KR)
Venue
IEEE Access
Published
2020
Volume
8
Pages
203814-203826
ISSN
2169-3536 (ISSN-L)
OpenAlex
W3101150053
Cited by
39
References
58
Citations by year
2026: 5 · 2025: 14 · 2024: 7 · 2023: 6 · 2022: 2 · 2021: 5
FWCI
3.14
Type
journal article
Language
en
Open access
Locations
  • IEEE Access (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Korea Evaluation Institute of Industrial Technology (10073154)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
  • Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
Keywords
multimodal fusion, bilinear pooling, low-rank decomposition, EEG features, human-computer interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.