Real-Time Video Emotion Recognition Based on Reinforcement Learning and Domain Knowledge
The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Abstract
Multimodal emotion recognition in conversational videos (ERC) develops rapidly in recent years. To fully extract the relative context from video clips, most studies build their models on the entire dialogues which make them lack of real-time ERC ability. Different from related researches, a novel multimodal emotion recognition model for conversational videos based on reinforcement learning and domain knowledge (ERLDK) is proposed in this paper. In ERLDK, the reinforcement learning algorithm is introduced to conduct real-time ERC with the occurrence of conversations. The collection of history utterances is composed as an emotion-pair which represents the multimodal context of the following utterance to be recognized. Dueling deep-Q-network (DDQN) based on gated recurrent unit (GRU) layers is designed to learn the correct action from the alternative emotion categories. Domain knowledge is extracted from public dataset based on the former information of emotion-pairs. The extracted domain knowledge is used to revise the results from the RL module and is transformed into other dataset to examine the rationality. The experimental results on datasets show that ERLDK achieves the state-of-the-art results on weighted average and most of the specific emotion categories.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“the increase in patterns in decision-level fusion means that multiple models must be designed, leading to inefficient multi-pattern recognition algorithms that are difficult to port to mobile devices and provide real-time detection in daily use [22].”
From Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG (2024) · cites this paper as [22]
Citations
Cites
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 6×
“Secondly, to test the performances of different sizes of sample window, experiments on IEMOCAP and MELD are carried out with total dialogues in testsets and the results are listed in Table II compared with the other results of baselines.”
MethodInfluentialTensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 2×
“TFN[58]: This multimodal method fuses information of”
Self-citationMELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., & Mihalcea, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1050 · Cited by 1,416 · cited 4×
“Our model is evaluated on two widely accepted public datasets, IEMOCAP[55] and MELD[56].”
MethodInfluentialSelf-citationIdentifying Stable Patterns over Time for Emotion Recognition from EEG
Zheng, W.-L., Zhu, J.-Y., & Lu, B.-L. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2712143 · Cited by 990 · cited 4×
“On the contrary, EEG signals can make up for these ambiguities by recording the ongoing neuronal activities of the brain.”
BackgroundInfluentialContext-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939 · cited 1×
“c-LSTM+Att[57]: To represent the efficient context information for the target utterance, utterances around the target are input to attention based bidiredectional LSTM at each time spot.”
Self-citationDialogueRNN: An Attentive RNN for Emotion Detection in Conversations
Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., & Cambria, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016818 · Cited by 860
Self-citationMemory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855
Self-citationDialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation
Ghosal, D., Majumder, N., Poria, S., Chhaya, N., & Gelbukh, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1015 · Cited by 717 · cited 1×
“Ghosal et al. [18] presents a Dialogue Graph Convolutional Network (DialogueGCN) model based on former DialogueRNN model.”
MethodConversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos
Hazarika, D., Poria, S., Zadeh, A., Cambria, E., Morency, L.-P., & Zimmermann, R. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1193 · Cited by 463
Self-citationICON: Interactive Conversational Memory Network for Multimodal Emotion Detection
Hazarika, D., Poria, S., Mihalcea, R., Cambria, E., & Zimmermann, R. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1280 · Cited by 452 · cited 1×
“ICON [60]: ICON is similar to CMN that separate memory networks are used for speakers.”
MethodSelf-citationVideo Summarization With Attention-Based Encoder–Decoder Networks
Ji, Z., Xiong, K., Pang, Y., & Li, X. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2904996 · Cited by 392 · cited 1×
“People are inclined to express their emotions implicitly through some habitual ways during interpersonal conversations, which are different from the official circumstances[4].”
BackgroundSelf-citationLearning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346 · cited 1×
“mislead the results without the text[36].”
Cross-Subject Emotion Recognition Using Flexible Analytic Wavelet Transform From EEG Signals
Gupta, V., Chopda, M. D., & Pachori, R. B. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2018.2883497 · Cited by 293 · cited 1×
“Gupta et al. [34] proposes an effective method based on flexible analytic wavelet transform (FAWT) for recognition of emotion through the investigation of the channel specific nature of EEG. EEG performs better accuracy in many cases, however, it is not easy to obtain which makes it hard to be…”
MethodPCC Net: Perspective Crowd Counting via Spatial Convolutional Network
Gao, J., Wang, Q., & Li, X. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2919139 · Cited by 284 · cited 1×
“Motivated by these challenges, a large number of methods and theories for ERC have emerged in recent years[6].”
BackgroundSelf-citationA Multiview-Based Parameter Free Framework for Group Detection
Li, X., Chen, M., Nie, F., & Wang, Q. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11208 · Cited by 178 · cited 1×
“Both audio and visual signals mentioned above are common and frequently used in daily life[32], however, these information are likely to be contradictory or deceptive when they are analyzed on their own[33].”
BackgroundSelf-citationLocality Adaptive Discriminant Analysis
Li, X., Chen, M., Nie, F., & Wang, Q. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/306 · Cited by 171 · cited 1×
“Text is one of the most frequently unimodal used for ERC due to its advantages of expressing the conversation information clearly and continuously[14], [15].”
BackgroundSelf-citationLocal Directional Ternary Pattern for Facial Expression Recognition
Ryu, B., Rivera, A. R., Kim, J., & Chae, O. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2726010 · Cited by 150 · cited 2×
“Emotion-related features are encoded by Ryu et al. [30] that the robustness of edge patterns in the edge region are taken into consider with the smooth regions.”
BackgroundSentiment Analysis and Topic Recognition in Video Transcriptions
Stappen, L., Baird, A., Cambria, E., & Schuller, B. W. (2021) · IEEE Intelligent Systems · doi:10.1109/mis.2021.3062200 · Cited by 132 · cited 1×
“Unlike audio signals in which tone-related information are paid mainly attention[25], visual pictures are more intuitive[26] and have better readability[27].”
BackgroundSelf-citationAn Active Learning Paradigm for Online Audio-Visual Emotion Recognition
Kansizoglou, I., Bampis, L., & Gasteratos, A. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2961089 · Cited by 121 · cited 1×
“Recognizing emotion using RL[46] and knowledge-”
Zhang, Y., Song, D., Li, X., Zhang, P., Wang, P., Rong, L., Yu, G., & Wang, B. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.04.003 · Cited by 120 · cited 2×
“Zhang et al.[40] builds a quantumlike multimodal network (QMN), which uses quantum theory (QT) and a LSTM network to analyze sentiment in conversa-”
BackgroundSelf-citationSpeech Emotion Recognition Using Deep Neural Network Considering Verbal and Nonverbal Speech Sounds
Huang, K.-Y., Wu, C.-H., Hong, Q.-B., Su, M.-H., & Chen, Y.-H. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682283 · Cited by 116 · cited 2×
“Huang et al.[24] focuses on nonverbal sounds which naturally exists in our daily conversation.”
BackgroundDeep Multi-View Feature Learning for Person Re-Identification
Tao, D., Guo, Y., Yu, B., Pang, J., & Yu, Z. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2726580 · Cited by 109 · cited 1×
“Deep features and handcraft features of multiple views are innovatively combined in a simple but effective way by Tao et al. [31] for person re-identification.”
BackgroundHeterogeneous Face Recognition: A Common Encoding Feature Discriminant Approach
Gong, D., Li, Z., Huang, W., Li, X., & Tao, D. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2651380 · Cited by 107 · cited 1×
“Both audio and visual signals mentioned above are common and frequently used in daily life[32], however, these information are likely to be contradictory or deceptive when they are analyzed on their own[33].”
BackgroundSelf-citationHybrid Approach for Emotion Classification of Audio Conversation Based on Text and Speech Mining
Bhaskar, J., Sruthi, K., & Nedungadi, P. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.02.112 · Cited by 106
Conversational transfer learning for emotion recognition
Hazarika, D., Poria, S., Zimmermann, R., & Mihalcea, R. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.06.005 · Cited by 99 · cited 1×
“Text is one of the most frequently unimodal used for ERC due to its advantages of expressing the conversation information clearly and continuously[14], [15].”
BackgroundAdapted Dynamic Memory Network for Emotion Recognition in Conversation
Xing, S., Mai, S., & Hu, H. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3005660 · Cited by 95 · cited 3×
“Xing et al. [11] proposes an Adapted Dynamic Memory Network (A-DMN) in which regards the self and cross-speaker influence as two mainly points to the emotion flows of conversation.”
BackgroundInfluentialDeep reinforcement learning for robust emotional classification in facial expression recognition
Li, H., & Xu, H. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106172 · Cited by 88 · cited 1×
“Li et al.[49] proposes a RL model for pre-selecting (RLPS) useful images for facial emotion recognition, which is made up of an image selector and a rough emotion classifier.”
BackgroundMultimodal Local-Global Attention Network for Affective Video Content Analysis
Ou, Y., Chen, Z., & Wu, F. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3014889 · Cited by 84 · cited 1×
“However, it is not easy for computer algorithms to give most appropriate considerations to all these three characteristics of ERC like humans[8].”
BackgroundTensor Rank Preserving Discriminant Analysis for Facial Recognition
Tao, D., Guo, Y., Li, Y., & Gao, X. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2762588 · Cited by 79 · cited 3×
“Tao et al. [28] proposes a two-stage module to find the low-dimensional tensor subspace and computing the spectral of the face tensors.”
BackgroundSelf-citationFace Sketch Synthesis by Multidomain Adversarial Learning
Zhang, S., Ji, R., Hu, J., Lu, X., & Li, X. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2869574 · Cited by 75 · cited 1×
“Unlike audio signals in which tone-related information are paid mainly attention[25], visual pictures are more intuitive[26] and have better readability[27].”
BackgroundSelf-citationAn Interaction-aware Attention Network for Speech Emotion Recognition in Spoken Dialogs
Yeh, S.-L., Lin, Y.-S., & Lee, C.-C. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683293 · Cited by 67 · cited 1×
“To capture the contextual emotional changes as much as possible during actual research processes, majority studies choose to build models on whole dialogue range[9] and extra layers are designed to track the flow of emotion”
BackgroundADRL: An attention-based deep reinforcement learning framework for knowledge graph reasoning
Wang, Q., Hao, Y., & Cao, J. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.105910 · Cited by 65 · cited 1×
“RL has its unique advantages on imitating the real-time conversion of the conversations[50] and the domain knowledge in conversations[51] is also meaningful because the emotion of speakers is relatively following regular emotion inertia[52].”
BackgroundImproved emotion recognition in Spanish social media through incorporation of lexical knowledge
Plaza-del-Arco, F. M., Valdivia, M. T. M., López, L. A. U., & Mitkov, R. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2019.09.034 · Cited by 59 · cited 2×
“Plaza et al. [42] recognizes emotions by integrating different affective lexical knowledge from Spanish social media with neural networks.”
BackgroundRobust Web Image Annotation via Exploring Multi-Facet and Structural Knowledge
Hu, M., Yang, Y., Shen, F., Zhang, L., Shen, H. T., & Li, X. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2717185 · Cited by 54 · cited 1×
“However, using unimodal for ERC is insufficient and unstable in several cases, for example, faces can be blocked[35] and audio signals may”
MethodSelf-citationALSTM: An attention-based long short-term memory framework for knowledge base reasoning
Wang, Q., & Hao, Y. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.02.065 · Cited by 53 · cited 1×
“base system[47] are not new things actually.”
Information-Driven Multirobot Behavior Adaptation to Emotional Intention in Human–Robot Interaction
Chen, L., Wu, M., Zhou, M., She, J., Dong, F., & Hirota, K. (2018) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2017.2728003 · Cited by 51 · cited 1×
“RL has its unique advantages on imitating the real-time conversion of the conversations[50] and the domain knowledge in conversations[51] is also meaningful because the emotion of speakers is relatively following regular emotion inertia[52].”
BackgroundDriver Drowsiness Recognition via 3D Conditional GAN and Two-Level Attention Bi-LSTM
Hu, Y., Lu, M., Xie, C., & Lu, X. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2958188 · Cited by 51 · cited 1×
“In order to better adapt to daily application scenarios, like smart home, mental illness care, education aids and car-hailing services[1], many recent studies are not satisfied with simply classifying a single utterance, sentence or article using traditional emotion categorization models.”
BackgroundReinforcement online learning for emotion prediction by using physiological signals
Liu, W., Zhang, L., Tao, D., & Cheng, J. S. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.06.004 · Cited by 50 · cited 2×
“Liu et al. [48] proposes a Reinforcement Online Learning (ROL) method for real-time emotion state prediction by using EEG.”
MethodInteractive double states emotion cell model for textual dialogue emotion prediction
Li, D., Li, Y., & Wang, S. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.105084 · Cited by 50 · cited 1×
“The unique characteristics of ERC are summarized as context dependence, persistence and contagiousness[7] that the static and dynamic flows in a daily dialogue are both included.”
BackgroundSelf-citationBrain-Inspired Motion Learning in Recurrent Neural Network With Emotion Modulation
Huang, X., Wu, W. L., Qiao, H., & Ji, Y. (2018) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2018.2843563 · Cited by 48 · cited 1×
“integrating RNN with other algorithms and other relative subjects knowledge [41].”
MethodLaughter and Smiling in 16 Positive Emotions
Hofmann, J., Platt, T., & Ruch, W. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2737000 · Cited by 42 · cited 2×
“Hofmann et al. [29] investigates the elicitation of smiling and laughter and the role of facial display regulation markers in positive emotions during conversations.”
BackgroundAnnotating and modeling empathy in spoken conversations
Alam, F., Danieli, M., & Riccardi, G. (2018) · Computer Speech & Language · doi:10.1016/j.csl.2017.12.003 · Cited by 41 · cited 1×
“Some other unimodal approaches ERC include works that focus on audio[19], visual[20] and electroencephalogram (EEG)[21].”
BackgroundDynamic emotion modelling and anomaly detection in conversation based on emotional transition tensor
Sun, X. N., Zhang, C., & Li, L. (2018) · Information Fusion · doi:10.1016/j.inffus.2018.04.001 · Cited by 41 · cited 1×
“by just transforming algorithms and results from original emotion recognition issues[3].”
BackgroundUnsupervised Video Action Clustering via Motion-Scene Interaction Constraint
Peng, B., Lei, J., Fu, H., Zhang, C., Chua, T., & Li, X. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2889514 · Cited by 39 · cited 1×
“of dialogues, especially conversational videos, has become increasingly popular [2].”
BackgroundSelf-citationHuman–machine dialogue modelling with the fusion of word- and sentence-level emotions
Peng, D., Zhou, M., Liu, C., & Ai, J. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.105319 · Cited by 37 · cited 2×
“Peng et al. [16] proposes a text based model which fuses the word-level and sentence-level features to learn emotional expression which focuses on human–machine dialogue systems.”
BackgroundEmotional editing constraint conversation content generation based on reinforcement learning
Sun, X. N., Li, J., Wei, X., Li, C., & Tao, J. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.10.007 · Cited by 34 · cited 1×
“However, few of them try to combine RL and domain knowledge in real-time ERC, even less using different modalities as inputs[53], [54].”
BackgroundLong-Short-Term Features for Dynamic Scene Classification
Huang, Y., Cao, X., Wang, Q., Zhang, B., Zhen, X., & Li, X. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2823360 · Cited by 32 · cited 1×
“Therefore, recognitions on article level or paragraph level are imprecisely and lack of meaning because of the too long span[5].”
BackgroundSelf-citationEmotional Communication in Advanced Pediatric Cancer Conversations
Sisk, B. A., Friedrich, A. B., DuBois, J. M., & Mack, J. W. (2019) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2019.11.005 · Cited by 32 · cited 2×
“Bryan et al. [23] utilizes its proposed model based on audio features into health psychology field.”
BackgroundA Dialogical Emotion Decoder for Speech Emotion Recognition in Spoken Dialog
Yeh, S.-L., Lin, Y.-S., & Lee, C.-C. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053561 · Cited by 29 · cited 1×
“RNN and RNN-like networks are mainstream algorithms due to their significant advantages in processing sequence information[39].”
BackgroundKnowledge-Augmented Multimodal Deep Regression Bayesian Networks for Emotion Video Tagging
Wang, S., Hao, L., & Ji, Q. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2934824 · Cited by 26
Linear Disentangled Representation Learning for Facial Actions
Xiang, X., & Tran, T. D. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2771150 · Cited by 24
Connecting Model-Based and Model-Free Control With Emotion Modulation in Learning Systems
Huang, X., Wu, W. L., & Qiao, H. (2019) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2019.2933152 · Cited by 23 · cited 1×
“RL has its unique advantages on imitating the real-time conversion of the conversations[50] and the domain knowledge in conversations[51] is also meaningful because the emotion of speakers is relatively following regular emotion inertia[52].”
BackgroundLearning blur invariant binary descriptor for face recognition
Zhao, C., Li, X., & Dong, Y. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.04.082 · Cited by 22 · cited 1×
“Unlike audio signals in which tone-related information are paid mainly attention[25], visual pictures are more intuitive[26] and have better readability[27].”
BackgroundSelf-citationSpeech Emotion Recognition using Convolutional Neural Network with Audio Word-based Embedding
Huang, K.-Y., Wu, C., Hong, Q.-B., Su, M.-H., & Zeng, Y.-R. (2018) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp.2018.8706610 · Cited by 20 · cited 1×
“fore, there are many benefits to use multimodal as the inputs[38].”
BackgroundHow can we use knowledge about the neurobiology of emotion recognition in practice?
Hunnikin, L. M., & van Goozen, S. H. M. (2018) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2018.01.005 · Cited by 18 · cited 1×
“However, few of them try to combine RL and domain knowledge in real-time ERC, even less using different modalities as inputs[53], [54].”
BackgroundIdentity-Preserving Face Hallucination via Deep Reinforcement Learning
Cheng, X., Lu, J., Yuan, B., & Zhou, J. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2961629 · Cited by 17 · cited 9×
“During training, the Q net of DDQN is trained to output the right probabilities q eval ( s ( t ) , a ( t )) of chosen action from the six alternative emotions, which is represented as a ( t ) ∈ A, A ction = [0 , 1 , 2 , 3 , 4 , 5] .”
MethodInfluentialDicé, F., Dolce, P., Maiello, A., & Freda, M. F. (2019) · Pratiques Psychologiques · doi:10.1016/j.prps.2018.12.001 · Cited by 14 · cited 1×
“Francesca et al. [22] transcribes and analyzes audio conversations with verona coding definitions of emotional sequences to establish a quantitative relationship between asymmetrical variables.”
BackgroundLai, H., Chen, H., & Wu, S. (2020) · IEEE Access · doi:10.1109/access.2020.3005664 · Cited by 13 · cited 2×
“Lai et al. [45] proposes a different contextual window sizes based recurrent neural networks (DCWS-RNNs) model.”
BackgroundSocial Signal Processing for Evaluating Conversations Using Emotion Analysis and Sentiment Detection
Singh, P. K., Pisipati, N., Krishna, P. R., & Prasad, M. V. N. K. (2019) · International Conference on Advanced Computational and Communication Paradigms (ICACCP) · doi:10.1109/icaccp.2019.8883004 · Cited by 4 · cited 1×
“The expression of human during conversations is the comprehensive result of multiple behavioral patterns[37], there-”
Background
Cited by
Middya, A. I., Nag, B., & Roy, S. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108580 · Cited by 235
Video sentiment analysis with bimodal information-augmented multi-head attention
Wu, T., Peng, J., Zhang, W., Zhang, H., Tan, S., Yi, F., Ma, C., & Huang, Y. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107676 · Cited by 131
“Other strategies involve interesting domains with novel ideas including reinforcement learning [2, 6, 33, 60], fuzzy logic [4], bilinear pooling [64], deep canonical correlation analysis [40], domain adaptation [37], topic recognition [39], modality distillation [34], quantum learning[20, 62, 63],…”
BackgroundKnowleNet: Knowledge fusion network for multimodal sarcasm detection
Yue, T., Mao, R., Wang, H., Hu, Z., & Cambria, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101921 · Cited by 171
“Some works [13, 14, 15, 16] argued that external knowledge is critical for detecting and understanding non-literal expressions.”
BackgroundSelf-citationZhu, X., Liu, Z., Cambria, E., Yu, X., Fan, X., Chen, H., & Wang, R. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108564 · Cited by 159
Self-citationMultimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG
Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., & Wang, S. (2024) · IEEE Open Journal of Engineering in Medicine and Biology · doi:10.1109/ojemb.2023.3240280 · Cited by 121
Metadata
- Authors
- Zhang, K., Li, Y., Wang, J., Cambria, E., & Li, X.
- Affiliations
- Northwestern Polytechnical University (CN)
- Nanyang Technological University (SG)
- Venue
- IEEE Transactions on Circuits and Systems for Video Technology
- Published
- 2022-03
- Volume
- 32
- Issue
- 3
- Pages
- 1034-1047
- ISSN
- 1051-8215 (ISSN-L)
- 1558-2205
- OpenAlex
- W3153807302
- Cited by
- 184
- References
- 59
- Citations by year
- 2026: 9 · 2025: 42 · 2024: 36 · 2023: 46 · 2022: 39
- FWCI
- 32.10
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Circuits and Systems for Video Technology (publishedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion)
- Funders
- National Natural Science Foundation of China (61871470, U1801262, 61976179)
- Fundamental Research Funds for the Central Universities (3102019HTXM005, 3102017HQZZ003)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- video-based emotion recognition, conversational video, multimodal emotion recognition, real-time emotion recognition, reinforcement learning, domain knowledge, dueling deep Q-network, gated recurrent unit, dialogue context modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.