The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.
A Hybrid Latent Space Data Fusion Method for Multimodal Emotion Recognition
Open accessAbstract
Multimodal emotion recognition is an emerging interdisciplinary field of research in the area of affective computing and sentiment analysis. It aims at exploiting the information carried by signals of different nature to make emotion recognition systems more accurate. This is achieved by employing a powerful multimodal fusion method. In this study, a hybrid multimodal data fusion method is proposed in which the audio and visual modalities are fused using a latent space linear map and then, their projected features into the cross-modal space are fused with the textual modality using a Dempster-Shafer (DS) theory-based evidential fusion method. The evaluation of the proposed method on the videos of the DEAP dataset shows its superiority over both decision-level and non-latent space fusion methods. Furthermore, the results reveal that employing Marginal Fisher Analysis (MFA) for feature-level audio-visual fusion results in higher improvement in comparison to cross-modal factor analysis (CFA) and canonical correlation analysis (CCA). Also, the implementation results show that exploiting textual users' comments with the audiovisual content of movies improves the performance of the system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Within the field of Music Emotion Recognition (MER), Nemati et al. [90] introduced a comparable hybrid fusion methodology. Initially, they mapped the speech and visual modalities onto a shared latent space using feature-level fusion, a technique similar to early fusion methodologies.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [90]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093 · cited 2×
“According to the previous studies [59], [62], [66], [68], [69], In addition to the color energy, for each n × m frame, the 16 histogram of hue in the HSV space is”
MethodA Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Survey on speech emotion recognition: Features, classification schemes, and databases
El Ayadi, M., Kamel, M. S., & Karray, F. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,073
A Generalization of Bayesian Inference
Dempster, A. P. (1968) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1968.tb00722.x · Cited by 2,013
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 1×
“A comprehensive review of bimodal facial expression-based emotion recognition and audio modality may be found in [1], [40]–[42].”
BackgroundA Generalization of Bayesian Inference
Dempster, A. P. (2008) · Studies in Fuzziness and Soft Computing · doi:10.1007/978-3-540-44792-4_4 · Cited by 1,702 · cited 3×
“In the current study, following the promissing results reported in [14], [14], [15], we used the Dempster-Shafer (DS) [29] fusion method for”
BackgroundMethodA review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · cited 5×
“It has been shown that multimodal emotion recognition usually outperforms unimodal systems [17], [19], [32].”
BackgroundInfluential(2008) · doi:10.1007/978-0-387-71041-9 · Cited by 1,406
Affective Computing and Sentiment Analysis
Cambria, E. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.31 · Cited by 1,204 · cited 2×
“Another difference is that the former improves the viewers’ content selection and consumption [12], while the latter reveals the affective states of opinion holders who record their opinions on products, events, or services [2], [19].”
BackgroundEmotions Recognition Using EEG Signals: A Survey
Alarcao, S. M., & Fonseca, M. J. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2714671 · Cited by 1,201 · cited 2×
“Alarcao and Fonseca [49] provided a good review of EEG-based emotion recognition systems.”
BackgroundSchuller, B., Batliner, A., Steidl, S., & Seppi, D. (2011) · Speech Communication · doi:10.1016/j.specom.2011.01.011 · Cited by 758 · cited 1×
“quency Cepstrum Coefficient (MFCC), Zero Crossing Rate (ZCR), energy, and pitch can be used to extract emotions from the audio channel [72]–[74].”
BackgroundMultimodal Emotion Recognition in Response to Videos
Soleymani, M., Pantic, M., & Pun, T. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.37 · Cited by 722 · cited 8×
“Different combinations of modalities are proposed in the literature, including the fusion of facial expression with speech [1], [8], audiovisual and physiological signals [12], and the EEG response with textual modality [16].”
BackgroundMethodInfluentialGeneralized Multiview Analysis: A discriminative latent space
Sharma, A., Kumar, A., Daume, H., & Jacobs, D. W. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247923 · Cited by 714 · cited 18×
“2 compares the correlation heat maps of audio, visual, concatenation of audio and visual, and the fusion of audio and video using CCA, CFA, and MFA methods.”
BackgroundMethodInfluentialA Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628 · cited 3×
“A comprehensive review of multimodal affect recognition and description of different combinations of modalities can be found in [32].”
BackgroundThe eNTERFACE’05 Audio-Visual Emotion Database
Martin, O., Kotsia, I., Macq, B., & Pitas, I. (2006) · 22nd International Conference on Data Engineering Workshops (ICDEW'06) · doi:10.1109/icdew.2006.145 · Cited by 615 · cited 1×
“To address this problem, later research studies such as [12] and [36] assessed their systems performance on standard multimodal databases such as eNTERFACE’05 [39] and RML [36].”
MethodASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors
Subramanian, R., Wache, J., Abadi, M. K., Vieriu, R. L., Winkler, S., & Sebe, N. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2625250 · Cited by 535 · cited 1×
“Exploiting internal affective content and external users’ comments is preferred over using physiological changes as it does not require any physiological sensors [24].”
BackgroundFusing audio, visual and textual clues for sentiment analysis from multimodal content
Poria, S., Cambria, E., Howard, N., Huang, G.-B., & Hussain, A. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.01.095 · Cited by 534 · cited 3×
“Recently, multimodal emotion recognition has attracted an increasing attention of researchers as it can overcome the limitations of monomodal systems [8]–[10].”
BackgroundMethodMethodologies for Cross-Domain Data Fusion: An Overview
Zheng, Y. (2015) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2015.2465959 · Cited by 454 · cited 1×
“Treating different modalities equally by concatenating their feature vector is shown to have low performance in data mining tasks [52].”
BackgroundEmotion recognition in human–computer interaction
Fragopanagos, N., & Taylor, J. G. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.006 · Cited by 414
Emotion Assessment From Physiological Signals for Adaptation of Game Difficulty
Chanel, G., Rebetez, C., Bétrancourt, M., & Pun, T. (2011) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/tsmca.2011.2116000 · Cited by 398 · cited 1×
“[45] fused peripheral physiological channel and EEG for adaptation of game difficulty according to the players emotions, and Wang et al.”
BackgroundMultimodal sentiment analysis using hierarchical fusion with context modeling
Majumder, N., Hazarika, D., Gelbukh, A., Cambria, E., & Poria, S. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.07.041 · Cited by 383
Xu, D., Yan, S., Tao, D., Lin, S., & Zhang, H.-J. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2007.906769 · Cited by 276 · cited 1×
“[58], applied MFA for content-based image retrieval (CBR).”
MethodAffective understanding in film
Wang, H. L., & Cheong, L.-F. (2006) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2006.873781 · Cited by 267 · cited 4×
“From the viewpoint of the viewer, the rapid changes in the scene indicate the dynamic and exciting nature of the scene [60].”
BackgroundMethodInfluentialAudio-Visual Emotion Recognition in Video Clips
Noroozi, F., Marjanovic, M., Njegus, A., Escalera, S., & Anbarjafari, G. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2713783 · Cited by 266 · cited 1×
“[50], proposed a convolutional neural network (CNN) model for summarizing key-frame videos which are then combined with audio modality in a late fusion/stacking fashion for emotion prediction.”
BackgroundMultimedia content processing through cross-modal association
Li, D., Dimitrova, N., Li, M., & Sethi, I. K. (2003) · Proceedings of the eleventh ACM international conference on Multimedia · doi:10.1145/957013.957143 · Cited by 265 · cited 12×
“2 compares the correlation heat maps of audio, visual, concatenation of audio and visual, and the fusion of audio and video using CCA, CFA, and MFA methods.”
MethodInfluentialCanonical Correlation Analysis for Data Fusion and Group Inferences
Correa, N., Adali, T., Li, Y.-O., & Calhoun, V. (2010) · IEEE Signal Processing Magazine · doi:10.1109/msp.2010.936725 · Cited by 264
Multimodal Saliency and Fusion for Movie Summarization Based on Aural, Visual, and Textual Attention
Evangelopoulos, G., Zlatintsi, A., Potamianos, A., Maragos, P., Rapantzikos, K., Skoumas, G., & Avrithis, Y. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2267205 · Cited by 263 · cited 2×
“The total signal energy of xt (n) is usually obtained as [20]: Energy = √√√√ 1 N N ∑”
BackgroundBlock Matching Algorithms for Motion Estimation
Zheng, L. (2005) · Tv Engineering · Cited by 261
Multimodal Sentiment Analysis: Addressing Key Issues and Setting Up the Baselines
Poria, S., Majumder, N., Hazarika, D., Cambria, E., Gelbukh, A., & Hussain, A. (2018) · IEEE Intelligent Systems · doi:10.1109/mis.2018.2882362 · Cited by 246 · cited 3×
“Another difference is that the former improves the viewers’ content selection and consumption [12], while the latter reveals the affective states of opinion holders who record their opinions on products, events, or services [2], [19].”
BackgroundMethodTowards an intelligent framework for multimodal affective data analysis
Poria, S., Cambria, E., Hussain, A., & Huang, G.-B. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.10.005 · Cited by 236
Multimodal emotion recognition using EEG and eye tracking data
Zheng, W.-L., Dong, B.-N., & Lu, B.-L. (2014) · 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/embc.2014.6944757 · Cited by 230 · cited 1×
“EEG signals have also been fused with facial expression [43], audio signals [47], and eye tracking data [48].”
BackgroundFusion of facial expressions and EEG for implicit affective tagging
Koelstra, S., & Patras, I. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.10.002 · Cited by 228
Human emotion recognition and analysis in response to audio music using brain signals
Bhatti, A. M., Majid, M., Anwar, S. M., & Khan, B. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.08.029 · Cited by 223 · cited 1×
“EEG signals have also been fused with facial expression [43], audio signals [47], and eye tracking data [48].”
BackgroundRecognizing Human Emotional State From Audiovisual Signals
Wang, Y., & Guan, L. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.921734 · Cited by 214
Emotion Recognition Based on Joint Visual and Audio Cues
Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.489 · Cited by 202 · cited 1×
“For example, in [34], [35], and [38], the authors evaluated their system on their own”
BackgroundRecognizing Human Emotional State From Audiovisual Signals*
Wang, Y., & Guan, L. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.927665 · Cited by 202 · cited 1×
“There is a long-standing research on the audio and visual content fusion [34]–[37].”
BackgroundDe Silva, L. C., & Ng, P. C. (2002) · Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) · doi:10.1109/afgr.2000.840655 · Cited by 196
On the use of computable features for film classification
Rasheed, Z., Sheikh, Y. A., & Shah, M. A. (2005) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2004.839993 · Cited by 187
Fusion of EEG response and sentiment analysis of products review to predict customer satisfaction
Kumar, S., Yadava, M., & Roy, P. P. (2018) · Information Fusion · doi:10.1016/j.inffus.2018.11.001 · Cited by 186 · cited 1×
“Different combinations of modalities are proposed in the literature, including the fusion of facial expression with speech [1], [8], audiovisual and physiological signals [12], and the EEG response with textual modality [16].”
BackgroundEmotion Recognition on Twitter: Comparative Study and Training a Unison Model
Colneric, N., & Demsar, J. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2807817 · Cited by 182 · cited 1×
“Different applications benefit from emotion recognition, including video games [3], military healthcare [4], tutoring systems [5], predicting customer satisfaction [6], and Twitter analysis [7].”
BackgroundAudiovisual Synchronization and Fusion Using Canonical Correlation Analysis
Sargin, M. E., Yemez, Y., Erzin, E., & Tekalp, A. M. (2007) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2007.906583 · Cited by 181 · cited 1×
“An early study on audiovisual fusion using CCA was proposed for open-set speaker identification [53].”
BackgroundAudiovisual emotion recognition in wild
Avots, E., Sapiński, T., Bachmann, M., & Kamińska, D. (2019) · Machine Vision and Applications · doi:10.1007/s00138-018-0960-9 · Cited by 160
Wang, Y., Guan, L., & Venetsanopoulos, A. N. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2012.2189550 · Cited by 155 · cited 3×
“They evaluated their method on the SAVEE, eNTERFACE’05, and RML databases.”
MethodAffective content detection using HMMs
Kang, H.-B. (2003) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/957013.957066 · Cited by 149 · cited 1×
“For example, it has been shown that yellow, orange, and red are associated with feelings of fear and discomfort, while blue, purple, and green can evoke feelings of high valence and low arousal in the viewer [63], [65], [67].”
BackgroundContinuous emotion detection using EEG signals and facial expressions
Soleymani, M., Asghari-Esfeden, S., Pantić, M., & Fu, Y. (2014) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2014.6890301 · Cited by 126 · cited 3×
“EEG signals have also been fused with facial expression [43], audio signals [47], and eye tracking data [48].”
BackgroundLin, Y., Yang, Y., & Jung, T. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00094 · Cited by 109 · cited 1×
“Low coefficients of this feature have been commonly used in past studies [62], [66], [72], [76]–[78].”
Yan, J., Zheng, W., Xu, Q., Lu, G., Li, H., & Wang, B. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2557721 · Cited by 106 · cited 2×
“Recently, multimodal emotion recognition has attracted an increasing attention of researchers as it can overcome the limitations of monomodal systems [8]–[10].”
BackgroundMultimedia content processing through cross-modal association
Li, D., Dimitrova, N., Li, M., & Sethi, I. K. (2003) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/957142.957143 · Cited by 106
Affective Visualization and Retrieval for Music Video
Zhang, S., Huang, Q., Jiang, S., Gao, W., & Tian, Q. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2059634 · Cited by 98 · cited 4×
“3) Pitch: It shows the basic frequency of the signal [63].”
BackgroundMethodInfluentialA supervised approach to movie emotion tracking
Malandrakis, N., Potamianos, A., Evangelopoulos, G., & Zlatintsi, A. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5946961 · Cited by 93 · cited 1×
“Low coefficients of this feature have been commonly used in past studies [62], [66], [72], [76]–[78].”
MethodA Combined Rule-Based & Machine Learning Audio-Visual Emotion Recognition Approach
Seng, K. P., Ang, L.-M., & Ooi, C. S. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2588488 · Cited by 92 · cited 3×
“Sensor-level and feature-level fusion methods belong to the first category, while scorelevel, rank-level, and decision-level methods are known as the fusion after matching methods [12], [13].”
BackgroundMethodAutomatic Recognition of Facial Displays of Unfelt Emotions
Kulkarni, K., Corneanu, C. A., Ofodile, I., Escalera, S., Baró, X., Hyniewska, S., Allïk, J., & Anbarjafari, G. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2874996 · Cited by 78 · cited 1×
“[51], proposed a CNN model trained for learning a static representation from images and enhancing it through Fisher vector encoding for visual feature extraction.”
MethodEmotion Recognition in Affective Tutoring Systems: Collection of Ground-truth Data
Petroviča, S., Anohina-Naumeca, A., & Ekenel, H. K. (2017) · Procedia Computer Science · doi:10.1016/j.procs.2017.01.157 · Cited by 74 · cited 1×
“Different applications benefit from emotion recognition, including video games [3], military healthcare [4], tutoring systems [5], predicting customer satisfaction [6], and Twitter analysis [7].”
BackgroundHybrid video emotional tagging using users’ EEG and video content
Wang, S., Zhu, Y., Wu, G., & Ji, Q. (2013) · Multimedia Tools and Applications · doi:10.1007/s11042-013-1450-8 · Cited by 72 · cited 1×
“[46] combined EEG signal and video content for emotional video tagging.”
BackgroundUnderstanding Affective Content of Music Videos through Learned Representations
Acar, E., Hopfgartner, F., & Albayrak, Ş. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-04114-8_26 · Cited by 67 · cited 3×
“According to the previous studies [59], [62], [66], [68], [69], In addition to the color energy, for each n × m frame, the 16 histogram of hue in the HSV space is”
BackgroundMethodAbdAlmageed, W. (2008) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2008.4712476 · Cited by 65 · cited 1×
“frames shows the shot length [62], [64].”
Multimedia content analysis for emotional characterization of music video clips
Yazdani, A., Skodras, E., Fakotakis, N. D., & Ebrahimi, T. (2013) · EURASIP Journal on Image and Video Processing · doi:10.1186/1687-5281-2013-26 · Cited by 65 · cited 5×
“frames shows the shot length [62], [64].”
BackgroundMethodInfluentialUsage of emotion recognition in military health care
Tokuno, S., Tsumatori, G., Shono, S., Eriko, T., Yamamoto, T., Suzuki, G., Mituyoshi, S., & Shimura, M. (2011) · Defense Science Research Conference and Expo (DSR) · doi:10.1109/dsr.2011.6026823 · Cited by 59 · cited 2×
“Different applications benefit from emotion recognition, including video games [3], military healthcare [4], tutoring systems [5], predicting customer satisfaction [6], and Twitter analysis [7].”
BackgroundMethodDynamic Probabilistic CCA for Analysis of Affective Behavior and Fusion of Continuous Annotations
Nicolaou, M. A., Pavlović, V., & Pantić, M. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.16 · Cited by 52 · cited 1×
“[54], used dynamic probabilistic CCA (DPCCA) to fuse continuous annotations for analyzing affective behavior.”
MethodThe Labeled Multiple Canonical Correlation Analysis for Information Fusion
Gao, L., Zhang, R., Qi, L., Chen, E., & Guan, L. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2859590 · Cited by 52 · cited 1×
“More recently, labeled multiple canonical correlation analysis (LMCCA) is proposed for bimodal human emotion recognition [56].”
BackgroundHierarchical affective content analysis in arousal and valence dimensions
Xu, M. G., Xu, C., He, X., Jin, J. S., Luo, S., & Rui, Y. (2012) · Signal Processing · doi:10.1016/j.sigpro.2012.06.026 · Cited by 50 · cited 3×
“Low-key with dark scenes is used to induce unpleasant emotions in violent and scary movies [65].”
BackgroundAffective content detection using HMMs
Kang, H.-B. (2003) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/957052.957066 · Cited by 49
Emotion Recognition for Affect Aware Video Games
Szwoch, M., & Szwoch, W. (2014) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-10662-5_28 · Cited by 48 · cited 1×
“Different applications benefit from emotion recognition, including video games [3], military healthcare [4], tutoring systems [5], predicting customer satisfaction [6], and Twitter analysis [7].”
BackgroundVideo Analytics for Customer Emotion and Satisfaction at Contact Centers
Seng, K. P., & Ang, L.-M. (2017) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2017.2695613 · Cited by 48 · cited 1×
“Different applications benefit from emotion recognition, including video games [3], military healthcare [4], tutoring systems [5], predicting customer satisfaction [6], and Twitter analysis [7].”
BackgroundSentiment Prediction Based on Dempster‐Shafer Theory of Evidence
Basiri, M. E., Naghsh‐Nilchi, A. R., & Ghasem-Aghaee, N. (2014) · Mathematical Problems in Engineering · doi:10.1155/2014/361201 · Cited by 47 · cited 2×
“After defining the evidence, we should define the mass function, a basic probability assignment (BPA) that must satisfy the following properties [30], [84]:”
BackgroundMethodSelf-citationExploiting reviewers’ comment histories for sentiment analysis
Basiri, M. E., Ghasem-Aghaee, N., & Naghsh‐Nilchi, A. R. (2014) · Journal of Information Science · doi:10.1177/0165551514522734 · Cited by 46 · cited 4×
“In the experiments, the following five evaluation criteria are used; Precision (π), recall (ρ), F-Measure, accuracy, and specifity [31], [55], [75], [83]:”
BackgroundMethodInfluentialSelf-citationEmotion recognition through integrating EEG and peripheral signals
Shu, Y., & Wang, S. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952681 · Cited by 46 · cited 1×
“The fusion of EEG signals with other modalities is also a prevalent combination [43], [44].”
BackgroundSun, K., & Yu, J. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_52 · Cited by 42 · cited 1×
“4) Motion: It has been shown that there is a relationship between camera movement and emotions such as happiness, sadness, and fear [70].”
BackgroundBasiri, M. E., & Kabiri, A. (2018) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3195633 · Cited by 41 · cited 2×
“Low-order N-grams including unigram and bigram are used broadly in NLP tasks to capture textual context [80]–[82].”
BackgroundMethodSelf-citationSentence-level sentiment analysis in Persian
Basiri, M. E., & Kabiri, A. (2017) · doi:10.1109/pria.2017.7983023 · Cited by 39
Self-citationHOMPer: A new hybrid system for opinion mining in the Persian language
Basiri, M. E., & Kabiri, A. (2019) · Journal of Information Science · doi:10.1177/0165551519827886 · Cited by 37 · cited 3×
“It has been shown that the combination of unigram and bigram features are effective for sentiment analysis [22].”
BackgroundSelf-citationAffective content analysis of music video clips
Yazdani, A., Kappeler, K., & Ebrahimi, T. (2011) · International ACM Workshop on Music Information Retrieval with User-Centered and Multimodal Strategies (MIRUM) · doi:10.1145/2072529.2072532 · Cited by 32 · cited 3×
“According to the previous studies [59], [62], [66], [68], [69], In addition to the color energy, for each n × m frame, the 16 histogram of hue in the HSV space is”
BackgroundMethodAn improved evidence-based aggregation method for sentiment analysis
Khiabani, P. J., Basiri, M. E., & Rastegari, H. (2019) · Journal of Information Science · doi:10.1177/0165551519837187 · Cited by 31 · cited 2×
“After defining the evidence, we should define the mass function, a basic probability assignment (BPA) that must satisfy the following properties [30], [84]:”
BackgroundMethodSelf-citationVisual/Acoustic Emotion Recognition
Chen, C., Huang, Y.-K., & Cook, P. R. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521709 · Cited by 28 · cited 2×
“There is a long-standing research on the audio and visual content fusion [34]–[37].”
BackgroundIncorporating social media comments in affective video retrieval
Nemati, S., & Naghsh‐Nilchi, A. R. (2015) · Journal of Information Science · doi:10.1177/0165551515593689 · Cited by 26 · cited 16×
“The first reported study that addressed the fusion of external textual modality with audiovisual content was presented by Nemati and Naghsh-Nilchi [14].”
BackgroundMethodInfluentialSelf-citationEyben, F., Weninger, F. J., Lehment, N. H., Schuller, B. W., & Rigoll, G. (2013) · PLoS ONE · doi:10.1371/journal.pone.0078506 · Cited by 26 · cited 1×
“According to the previous studies [59], [62], [66], [68], [69], In addition to the color energy, for each n × m frame, the 16 histogram of hue in the HSV space is”
MethodFF-SKPCCA: Kernel probabilistic canonical correlation analysis
Sarvestani, R. R., & Boostani, R. (2016) · Applied Intelligence · doi:10.1007/s10489-016-0823-x · Cited by 25 · cited 10×
“Such features may reduce the quality of emotion recognition systems if they are used in training the model [25].”
BackgroundMethodInfluentialNemati, S., & Naghsh‐Nilchi, A. R. (2017) · doi:10.1109/pria.2017.7983051 · Cited by 25 · cited 10×
“The mass function should be defined in a way that it shows the amount of support of evidence for each subset A ⊆ θ [9], [14].”
BackgroundMethodInfluentialSelf-citationTranslation is not enough: Comparing Lexicon-based methods for sentiment analysis in Persian
Basiri, M. E., & Kabiri, A. (2017) · International Symposium on Computer Science and Software Engineering (CSSE) · doi:10.1109/csicsse.2017.8320114 · Cited by 24 · cited 1×
“Low-order N-grams including unigram and bigram are used broadly in NLP tasks to capture textual context [80]–[82].”
MethodSelf-citationAutoencoder for Semisupervised Multiple Emotion Detection of Conversation Transcripts
Phan, D. A., Matsumoto, Y., & Shindo, H. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2885304 · Cited by 23 · cited 1×
“Textual modality may be in the form of either the movie transcript [20], [21] or the user comments in social networks [14], [15], [22].”
BackgroundParticle Swarm Optimization for Feature Selection in Speaker Verification
Nemati, S., & Basiri, M. E. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-12239-2_39 · Cited by 20 · cited 3×
“where its maximum is on the pitch [15], [75].”
BackgroundMethodSelf-citationAn evidential data fusion method for affective music video retrieval
Nemati, S., & Naghsh‐Nilchi, A. R. (2017) · Intelligent Data Analysis · doi:10.3233/ida-160029 · Cited by 20 · cited 10×
“Textual modality may be in the form of either the movie transcript [20], [21] or the user comments in social networks [14], [15], [22].”
BackgroundMethodInfluentialSelf-citationPuthenputhussery, A., Liu, Q., & Liu, C. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2685179 · Cited by 20 · cited 3×
“Latent Dirichlet Analysis (LDA) obtains the feature space by assuming the Gaussian distribution of the data in each class, which is not the case in many real-world problems, making the method ineffective [57].”
BackgroundMethodInfluentialCanonical Correlation Analysis for Data Fusion in Multimodal Emotion Recognition
Nemati, S. (2018) · International Symposium on Telecommunications · doi:10.1109/istel.2018.8661140 · Cited by 15 · cited 3×
“Recently, the original CCA method is used to fuse different modalities for multimodal emotion recognition [55].”
MethodSelf-citationUninorm operators for sentence-level score aggregation in sentiment analysis
Basiri, M. E., & Kabiri, A. (2018) · International Conference on Web Research (ICWR) · doi:10.1109/icwr.2018.8387244 · Cited by 9 · cited 3×
“In the experiments, the following five evaluation criteria are used; Precision (π), recall (ρ), F-Measure, accuracy, and specifity [31], [55], [75], [83]:”
MethodSelf-citationAn approach for exploring a video via multimodal feature extraction and user interactions
Salim, F. A., Haider, F., Conlan, O., & Luz, S. F. (2018) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-018-0268-0 · Cited by 8 · cited 1×
“Among these challenges, selecting the best combination of modalities has been the focus of several studies in recent years [9], [12], [17], [33].”
BackgroundOWA Operators for the Fusion of Social Networks’ Comments with Audio-Visual Content
Nemati, S. (2019) · International Conference on Web Research (ICWR) · doi:10.1109/icwr.2019.8765263 · Cited by 4
Self-citation
Research works printed but not linked: 1
- morris, 2006, Emotion and Anxiety A Philosophic Inquiry
Cited by
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
“, multi-modal Bayesian fusion [90], multi-level fusion [471], image fusion [472], data fusion [473], etc.”
BackgroundSelf-citationAbdar, M., Samami, M., Dehghani Mahmoodabad, S., Doan, T., Mazoure, B., Hashemifesharaki, R., Liu, L., Khosravi, A., Acharya, U. R., Makarenkov, V., et al. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104418 · Cited by 237
Self-citationA Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“Within the field of Music Emotion Recognition (MER), Nemati et al. [90] introduced a comparable hybrid fusion methodology.”
MethodCross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion
Cimtay, Y., Ekmekcioglu, E., & Caglar-Ozhan, S. (2020) · IEEE Access · doi:10.1109/access.2020.3023871 · Cited by 200
“The literature about emotional data fusion involves three data fusion techniques, which are early fusion (feature fusion) [42], [43], late fusion (decision fusion) [44]–[46] and hybrid approaches [17], [47], [48].”
BackgroundMethodFoundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L.-P. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 195
“Fine-grained fission: Beyond factorizing only into individual modality representations, fine-grained fission attempts to further break multimodal data down into the individual subspaces covered by the modalities [353].”
Background
Metadata
- Authors
- Nemati, S., Rohani, R., Basiri, M. E., Abdar, M., Yen, N. Y., & Makarenkov, V.
- Affiliations
- Shahrekord University (IR)
- Université du Québec à Montréal (CA)
- University of Aizu (JP)
- Venue
- IEEE Access
- Published
- 2019
- Volume
- 7
- Pages
- 172948-172964
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2994718079
- Cited by
- 98
- References
- 88
- Citations by year
- 2026: 8 · 2025: 22 · 2024: 13 · 2023: 23 · 2022: 14 · 2021: 9 · 2020: 9
- FWCI
- 9.43
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Shahrekord University (97GRN1M44264)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal emotion recognition, Dempster-Shafer theory, evidential fusion, audiovisual integration, marginal Fisher analysis, cross-modal fusion, DEAP dataset, online comments, affective computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.