The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Real-Time Video Emotion Recognition Based on Reinforcement Learning and Domain Knowledge
Abstract
Multimodal emotion recognition in conversational videos (ERC) develops rapidly in recent years. To fully extract the relative context from video clips, most studies build their models on the entire dialogues which make them lack of real-time ERC ability. Different from related researches, a novel multimodal emotion recognition model for conversational videos based on reinforcement learning and domain knowledge (ERLDK) is proposed in this paper. In ERLDK, the reinforcement learning algorithm is introduced to conduct real-time ERC with the occurrence of conversations. The collection of history utterances is composed as an emotion-pair which represents the multimodal context of the following utterance to be recognized. Dueling deep-Q-network (DDQN) based on gated recurrent unit (GRU) layers is designed to learn the correct action from the alternative emotion categories. Domain knowledge is extracted from public dataset based on the former information of emotion-pairs. The extracted domain knowledge is used to revise the results from the RL module and is transformed into other dataset to examine the rationality. The experimental results on datasets show that ERLDK achieves the state-of-the-art results on weighted average and most of the specific emotion categories.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“the increase in patterns in decision-level fusion means that multiple models must be designed, leading to inefficient multi-pattern recognition algorithms that are difficult to port to mobile devices and provide real-time detection in daily use [22].”
From Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG (2024) · cites this paper as [22]
Citations
Cites
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 6×
“Secondly, to test the performances of different sizes of sample window, experiments on IEMOCAP and MELD are carried out with total dialogues in testsets and the results are listed in Table II compared with the other results of baselines.”
MethodInfluentialTensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 2×
“TFN[58]: This multimodal method fuses information of”
Self-citationMELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., & Mihalcea, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1050 · Cited by 1,416 · cited 4×
“Our model is evaluated on two widely accepted public datasets, IEMOCAP[55] and MELD[56].”
MethodInfluentialSelf-citationIdentifying Stable Patterns over Time for Emotion Recognition from EEG
Zheng, W.-L., Zhu, J.-Y., & Lu, B.-L. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2712143 · Cited by 990 · cited 4×
“On the contrary, EEG signals can make up for these ambiguities by recording the ongoing neuronal activities of the brain.”
BackgroundInfluentialContext-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939 · cited 1×
“c-LSTM+Att[57]: To represent the efficient context information for the target utterance, utterances around the target are input to attention based bidiredectional LSTM at each time spot.”
Self-citationDialogueRNN: An Attentive RNN for Emotion Detection in Conversations
Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., & Cambria, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016818 · Cited by 860
Self-citationMemory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855
Self-citationDialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation
Ghosal, D., Majumder, N., Poria, S., Chhaya, N., & Gelbukh, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1015 · Cited by 717 · cited 1×
“Ghosal et al. [18] presents a Dialogue Graph Convolutional Network (DialogueGCN) model based on former DialogueRNN model.”
MethodConversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos
Hazarika, D., Poria, S., Zadeh, A., Cambria, E., Morency, L.-P., & Zimmermann, R. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1193 · Cited by 463
Self-citationICON: Interactive Conversational Memory Network for Multimodal Emotion Detection
Hazarika, D., Poria, S., Mihalcea, R., Cambria, E., & Zimmermann, R. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1280 · Cited by 452 · cited 1×
“ICON [60]: ICON is similar to CMN that separate memory networks are used for speakers.”
MethodSelf-citation
Cited by
Middya, A. I., Nag, B., & Roy, S. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108580 · Cited by 235
Video sentiment analysis with bimodal information-augmented multi-head attention
Wu, T., Peng, J., Zhang, W., Zhang, H., Tan, S., Yi, F., Ma, C., & Huang, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107676 · Cited by 145
“Other strategies involve interesting domains with novel ideas including reinforcement learning [2, 6, 33, 60], fuzzy logic [4], bilinear pooling [64], deep canonical correlation analysis [40], domain adaptation [37], topic recognition [39], modality distillation [34], quantum learning[20, 62, 63],…”
BackgroundKnowleNet: Knowledge fusion network for multimodal sarcasm detection
Yue, T., Mao, R., Wang, H., Hu, Z., & Cambria, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101921 · Cited by 171
“Some works [13, 14, 15, 16] argued that external knowledge is critical for detecting and understanding non-literal expressions.”
BackgroundSelf-citationZhu, X., Liu, Z., Cambria, E., Yu, X., Fan, X., Chen, H., & Wang, R. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108564 · Cited by 159
Self-citationMultimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG
Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., & Wang, S. (2024) · IEEE Open Journal of Engineering in Medicine and Biology · doi:10.1109/ojemb.2023.3240280 · Cited by 121
Metadata
- Authors
- Zhang, K., Li, Y., Wang, J., Cambria, E., & Li, X.
- Affiliations
- Northwestern Polytechnical University (CN)
- Nanyang Technological University (SG)
- Venue
- IEEE Transactions on Circuits and Systems for Video Technology
- Published
- 2022-03
- Volume
- 32
- Issue
- 3
- Pages
- 1034-1047
- ISSN
- 1051-8215 (ISSN-L)
- 1558-2205
- OpenAlex
- W3153807302
- Cited by
- 184
- References
- 59
- Citations by year
- 2026: 9 · 2025: 42 · 2024: 36 · 2023: 46 · 2022: 39
- FWCI
- 32.10
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Circuits and Systems for Video Technology (publishedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion)
- Funders
- National Natural Science Foundation of China (61871470, U1801262, 61976179)
- Fundamental Research Funds for the Central Universities (3102019HTXM005, 3102017HQZZ003)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- video-based emotion recognition, conversational video, multimodal emotion recognition, real-time emotion recognition, reinforcement learning, domain knowledge, dueling deep Q-network, gated recurrent unit, dialogue context modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Real-Time Video Emotion Recognition Based on Reinforcement Learning and Domain Knowledge. https://app.cochat.ai/papers/10.1109/tcsvt.2021.3072412
- The paper
- Zhang, K., Li, Y., Wang, J., Cambria, E., & Li, X. (2022). Real-Time Video Emotion Recognition Based on Reinforcement Learning and Domain Knowledge. IEEE Transactions on Circuits and Systems for Video Technology. https://doi.org/10.1109/tcsvt.2021.3072412
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.