CoChat

Information Fusion in Attention Networks Using Adaptive and Multi-Level Factorized Bilinear Pooling for Audio-Visual Emotion Recognition

Zhou, Hengshun, Du, Jun, Zhang, Yuanyuan, Wang, Qing, Liu, Qing-Feng, Lee, Chin-Hui · 2021 · IEEE/ACM Transactions on Audio, Speech, and Language Processing

The DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index.

Abstract

Multimodal emotion recognition is a challenging task in emotion computing as it is quite difficult to extract discriminative features to identify the subtle differences in human emotions with abstract concept and multiple expressions. Moreover, how to fully utilize both audio and visual information is still an open problem. In this paper, we propose a novel multimodal fusion attention network for audio-visual emotion recognition based on adaptive and multi-level factorized bilinear pooling (FBP). First, for the audio stream, a fully convolutional network (FCN) equipped with 1-D attention mechanism and local response normalization is designed for speech emotion recognition. Next, a global FBP (G-FBP) approach is presented to perform audio-visual information fusion by integrating self-attention based video stream with the proposed audio stream. To improve G-FBP, an adaptive strategy (AG-FBP) to dynamically calculate the fusion weight of two modalities is devised based on the emotion-related representation vectors from the attention mechanism of respective modalities. Finally, to fully utilize the local emotion information, adaptive and multi-level FBP (AM-FBP) is introduced by combining both global-trunk and intra-trunk data in one recording on top of AG-FBP. Tested on the IEMOCAP corpus for speech emotion recognition with only audio stream, the new FCN method outperforms the state-of-the-art results with an accuracy of 71.40%. Moreover, validated on the AFEW database of EmotiW2019 sub-challenge and the IEMOCAP corpus for audio-visual emotion recognition, the proposed AM-FBP approach achieves the best accuracy of 63.09% and 75.49% respectively on the test set.

Overview

Integrity signals

Registry

Crossref · 10.1109/taslp.2021.3096037

Cites

77 of 77 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 77 references

Research works printed but not linked: 1

  1. mahesh, 2019, Multimodal emotion recognition using facial expressions, body gestures, speech, and text modalities, Int J Eng Adv Technol, 8, 2453

Cited by

See all 103

Metadata

Authors
Zhou, H., Du, J., Zhang, Y., Wang, Q., Liu, Q.-F., & Lee, C.-H.
Affiliations
  • University of Science and Technology of China (CN)
  • Georgia Institute of Technology (US)
Venue
IEEE/ACM Transactions on Audio, Speech, and Language Processing
Published
2021
Volume
29
Pages
2617-2629
ISSN
  • 2329-9290 (ISSN-L)
  • 2329-9304
OpenAlex
W3186351311
Cited by
85
References
76
Citations by year
2026: 12 · 2025: 19 · 2024: 26 · 2023: 17 · 2022: 7 · 2021: 4
FWCI
13.05
Type
journal article
Language
en
Open access
closed
Locations
IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
Funders
  • Strategic Priority Research Program of Chinese Academy of Sciences (XDC08050200)
  • Ximalaya Inc.
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
audio-visual emotion recognition, multimodal fusion, attentional networks, adaptive fusion weights, self-attention, speech emotion recognition, fully convolutional network, IEMOCAP

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Information Fusion in Attention Networks Using Adaptive and Multi-Level Factorized Bilinear Pooling for Audio-Visual Emotion Recognition. https://app.cochat.ai/papers/10.1109/taslp.2021.3096037
The paper
Zhou, H., Du, J., Zhang, Y., Wang, Q., Liu, Q.-F., & Lee, C.-H. (2021). Information Fusion in Attention Networks Using Adaptive and Multi-Level Factorized Bilinear Pooling for Audio-Visual Emotion Recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing. https://doi.org/10.1109/taslp.2021.3096037

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.