A Hybrid Latent Space Data Fusion Method for Multimodal Emotion Recognition
Open accessThe DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.
Abstract
Multimodal emotion recognition is an emerging interdisciplinary field of research in the area of affective computing and sentiment analysis. It aims at exploiting the information carried by signals of different nature to make emotion recognition systems more accurate. This is achieved by employing a powerful multimodal fusion method. In this study, a hybrid multimodal data fusion method is proposed in which the audio and visual modalities are fused using a latent space linear map and then, their projected features into the cross-modal space are fused with the textual modality using a Dempster-Shafer (DS) theory-based evidential fusion method. The evaluation of the proposed method on the videos of the DEAP dataset shows its superiority over both decision-level and non-latent space fusion methods. Furthermore, the results reveal that employing Marginal Fisher Analysis (MFA) for feature-level audio-visual fusion results in higher improvement in comparison to cross-modal factor analysis (CFA) and canonical correlation analysis (CCA). Also, the implementation results show that exploiting textual users' comments with the audiovisual content of movies improves the performance of the system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Within the field of Music Emotion Recognition (MER), Nemati et al. [90] introduced a comparable hybrid fusion methodology. Initially, they mapped the speech and visual modalities onto a shared latent space using feature-level fusion, a technique similar to early fusion methodologies.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [90]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093 · cited 2×
“According to the previous studies [59], [62], [66], [68], [69], In addition to the color energy, for each n × m frame, the 16 histogram of hue in the HSV space is”
MethodA Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Survey on speech emotion recognition: Features, classification schemes, and databases
El Ayadi, M., Kamel, M. S., & Karray, F. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,073
A Generalization of Bayesian Inference
Dempster, A. P. (1968) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1968.tb00722.x · Cited by 2,013
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 1×
“A comprehensive review of bimodal facial expression-based emotion recognition and audio modality may be found in [1], [40]–[42].”
BackgroundA Generalization of Bayesian Inference
Dempster, A. P. (2008) · Studies in Fuzziness and Soft Computing · doi:10.1007/978-3-540-44792-4_4 · Cited by 1,702 · cited 3×
“In the current study, following the promissing results reported in [14], [14], [15], we used the Dempster-Shafer (DS) [29] fusion method for”
BackgroundMethodA review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · cited 5×
“It has been shown that multimodal emotion recognition usually outperforms unimodal systems [17], [19], [32].”
BackgroundInfluential(2008) · doi:10.1007/978-0-387-71041-9 · Cited by 1,406
Affective Computing and Sentiment Analysis
Cambria, E. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.31 · Cited by 1,204 · cited 2×
“Another difference is that the former improves the viewers’ content selection and consumption [12], while the latter reveals the affective states of opinion holders who record their opinions on products, events, or services [2], [19].”
Background
Research works printed but not linked: 1
- morris, 2006, Emotion and Anxiety A Philosophic Inquiry
Cited by
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
“, multi-modal Bayesian fusion [90], multi-level fusion [471], image fusion [472], data fusion [473], etc.”
BackgroundSelf-citationAbdar, M., Samami, M., Dehghani Mahmoodabad, S., Doan, T., Mazoure, B., Hashemifesharaki, R., Liu, L., Khosravi, A., Acharya, U. R., Makarenkov, V., et al. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104418 · Cited by 237
Self-citationA Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“Within the field of Music Emotion Recognition (MER), Nemati et al. [90] introduced a comparable hybrid fusion methodology.”
MethodCross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion
Cimtay, Y., Ekmekcioglu, E., & Caglar-Ozhan, S. (2020) · IEEE Access · doi:10.1109/access.2020.3023871 · Cited by 200
“The literature about emotional data fusion involves three data fusion techniques, which are early fusion (feature fusion) [42], [43], late fusion (decision fusion) [44]–[46] and hybrid approaches [17], [47], [48].”
BackgroundMethodFoundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L.-P. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 195
“Fine-grained fission: Beyond factorizing only into individual modality representations, fine-grained fission attempts to further break multimodal data down into the individual subspaces covered by the modalities [353].”
BackgroundEmotion recognition from unimodal to multimodal analysis: A review
Ezzameli, K., & Mahersia, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101847 · Cited by 187
A novel fusion-based deep learning model for sentiment analysis of COVID-19 tweets
Basiri, M. E., Nemati, S., Abdar, M., Asadi, S., & Acharrya, U. R. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107242 · Cited by 187
“It has been studied in different languages [30–34] and performed on the comments of different social media [35–37].”
Self-citationMultimodal sentimental analysis for social media applications: A comprehensive review
Chandrasekaran, G., Nguyen, T. N., & Hemanth D., J. (2021) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1415 · Cited by 159
Basiri, M. E., Abdar, M., Cifci, M. A., Nemati, S., & Acharya, U. R. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.105949 · Cited by 151
Self-citationAbdar, M., Fahami, M. A., Rundo, L., Radeva, P. I., Frangi, A. F., Acharya, U. R., Khosravi, A., Lam, H., Jung, A., & Nahavandi, S. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3168887 · Cited by 84
“Different fusion approaches, therefore, have been proposed to deal with this issue [11], [12], [35]–[38].”
Self-citationEarly depression detection in social media based on deep learning and underlying emotions
Figuerêdo, J. S. L., Maia, A. L. L. M., & Calumby, R. T. (2022) · Online Social Networks and Media · doi:10.1016/j.osnem.2022.100225 · Cited by 76
Zhang, Y., Cheng, C., Wang, S., & Xia, T. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.103877 · Cited by 63
Khan, U. A., Xu, Q., Liu, Y. M., Lagstedt, A., Alamäki, A. V., & Kauttonen, J. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01302-2 · Cited by 85
Joint multimodal sentiment analysis based on information relevance
Chen, D., Su, W., Wu, P., & Hua, B. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103193 · Cited by 67
Video-Based Cross-Modal Auxiliary Network for Multimodal Sentiment Analysis
Chen, R., Zhou, W., Li, Y., & Zhou, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3197420 · Cited by 53
“[31] investigated the advantages of latent space linear maps and designed a hybrid multimodal data fusion model for audiovisual features fusion.”
BackgroundSpeech Expression Multimodal Emotion Recognition Based on Deep Belief Network
Liu, D., Chen, L., Wang, Z., & Diao, G. (2021) · Journal of Grid Computing · doi:10.1007/s10723-021-09564-0 · Cited by 38
“plays an important role in human-computer interaction and intelligent computers [2].”
BackgroundSingh, P., Tripathi, M. K., Patil, M. B., Shivendra, & Neelakantappa, M. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19171-2 · Cited by 38
“Additionally, by concatenating feature vectors from many modalities to create a high dimensional feature vector, implicit association between modalities is ignored [19, 20].”
Learning Better Representations for Audio-Visual Emotion Recognition with Common Information
Ma, F., Zhang, W., Li, Y., Huang, S., & Zhang, L. (2020) · Applied Sciences · doi:10.3390/app10207239 · Cited by 38
Multimodal Emotion Recognition Using Bi-LG-GCN for MELD Dataset
Al-Saadawi, H. F. T., & Daş, R. (2024) · Balkan Journal of Electrical and Computer Engineering · doi:10.17694/bajece.1372107 · Cited by 36
“According to Nemati et al. [32] they suggested an integrated multimodal data fusion approach in which the audio and visual models were fused using a latent space linear map, and their presented characteristics into the cross-modal space were fused with the textual modality using a Dempster-Shafer…”
MethodZhang, D., Li, S., Zhu, Q., & Zhou, G. (2020) · IEEE Access · doi:10.1109/access.2020.2969205 · Cited by 28
“On the one hand, with regard to multi-modal data, no matter whether for supervised or semi-supervised learning, we should effectively capture the independent knowledge within each modality [35].”
BackgroundAspect-level multimodal sentiment analysis based on co-attention fusion
Wang, S., Cai, G., & Lv, G. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-023-00497-3 · Cited by 29
“In general, the existing multimodal fusion methods include feature-level fusion [5, 6], decision-level fusion [7, 8] and consistency regression fusion [9, 10].”
Multimodal Emotion Recognition Using Different Fusion Techniques
Subramanian, G., Cholendiran, N., Prathyusha, K., Balasubramanain, N., & Aravinth, J. (2021) · International Conference on Bio Signals, Images, and Instrumentation (ICBSII) · doi:10.1109/icbsii51839.2021.9445146 · Cited by 32
“Similarly Nemati et al. [13] studied, a hybrid fusion method for multimodal emotion recognition in which the audio and visual modalities are fused using a latent space linear map and then, their projected features were fused with the textual modality.”
MethodHe, X., Fang, Y., Xiang, N., Li, Z., Wang, Q., Yang, C., Wang, H. H., & Pan, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129532 · Cited by 39
“The existing multi-modal fusion methods usually include feature level fusion[9][10], decision level fusion[11][12] and consistent regression fusion[13][14].”
Liang, Y., Chen, G., Li, S., & Xu, Z. (2023) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-5460 · Cited by 26
A Framework to Evaluate Fusion Methods for Multimodal Emotion Recognition
Peña, D., Aguilera, A., Dongo, I., Heredia, J., & Cardinale, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3240420 · Cited by 34
“In [41] the audio and visual modalities are fused using a latent space linear map and then, their projected features into the cross-modal space are fused with the textual modality using a Dempster-Shafer (DS) theory-based evidential fusion method.”
MethodAziz, A., Chowdhury, N. K., Kabir, M. A., Chy, A. N., & Siddique, M. J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129376 · Cited by 30
Fan, L., Wang, F., Zhao, J., Zhang, J., Li, Y.-A., Tang, J. Y., Lin, T., & Wei, Q. (2025) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-025-01706-x · Cited by 24
Hybrid multi-modal emotion recognition framework based on InceptionV3DenseNet
Alamgir, F. M., & Alam, M. S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15066-w · Cited by 24
“The following steps from [22] are utilized in our framework to analyze the cepstral coefficients.”
MethodSentiment Analysis of Persian Instagram Post: a Multimodal Deep Learning Approach
Karimvand, A. N., Chegeni, R. S., Basiri, M. E., & Nemati, S. (2021) · International Conference on Web Research (ICWR) · doi:10.1109/icwr51868.2021.9443026 · Cited by 19
“Moreover, in some applications [6], [22], the modalities are not synchronized and hence cannot be combined using common feature-level fusion methods.”
BackgroundMethodSelf-citationMultimodal malware classification using proposed ensemble deep neural network framework
Nazim, S., Alam, M. M., Rizvi, S. S. A., Mustapha, J. C., Hussain, S. S., & Su’ud, M. M. (2025) · Scientific Reports · doi:10.1038/s41598-025-96203-3 · Cited by 27
“Another benefit of the late fusion methodology is that it employs the best techniques to evaluate each modality, such as the support vector machine (SVM) for images and hidden Markov models (HMM) for audio 33 .”
Gong, X., Dong, Y., & Zhang, T. (2023) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-023-01964-w · Cited by 25
“Nemati et al. [45] combined speech and video features at the feature layer and mixed text information at the decision-making layer to achieve hybrid fusion.”
Affective Interaction: Attentive Representation Learning for Multi-Modal Sentiment Classification
Zhang, Y., Tiwari, P., Rong, L., Chen, R., Alnajem, N. A., & Hossain, M. S. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3527175 · Cited by 21
“[33] proposed a hybrid multi-modal data fusion method, which used a latent space linear mapping function and feature concatenation.”
MethodThe Multimodal Emotion Information Analysis of E-Commerce Online Pricing in Electronic Word of Mouth
Chen, J., Zhong, Z., Feng, Q., & Liu, L. (2022) · Journal of Global Information Management · doi:10.4018/jgim.315322 · Cited by 20
“The Long Short-Term Memory (LSTM) of the long sequence of image frames obtained after classification has apparent advantages in the processing process (Nemati et al., 2019).”
BackgroundResearch on Extraction of Useful Tourism Online Reviews Based on Multimodal Feature Fusion
Li, M. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3453694 · Cited by 17
“…authors used the mentioned component analysis method to fuse the cortical macro-structure and used the univariate and machine learning approaches to assess the relationship between age, sex, case–control status [17], and symptom loads for depression and anxiety with the resulting brain components.”
MethodMultimodal Emotional Classification Based on Meaningful Learning
Filali, H., Riffi, J., Boulealam, C., Mahraz, M. A., & Tairi, H. (2022) · Big Data and Cognitive Computing · doi:10.3390/bdcc6030095 · Cited by 15
“In this work [32], the authors proposed a deep hierarchical architecture for modality fusion and applied it to the problem of sentiment analysis from the audio and text modalities.”
BackgroundChen, J., Li, Q., Liu, F., & Wen, Y. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109237 · Cited by 17
Devi, C. A., & Renuka, D. K. (2023) · IETE Journal of Research · doi:10.1080/03772063.2023.2173668 · Cited by 12
“Affective Computing is a field of artificial intelligence that enables the system to recognize human emotions [1,2].”
BackgroundKapase·, A. B., & Uke, N. (2025) · International Journal of Speech Technology · doi:10.1007/s10772-025-10202-3 · Cited by 22
Cross-Modal Learning for Audio-Visual Emotion Recognition in Acted Speech
Bouali, Y. L., Ahmed, O. B., & Mazouzi, S. (2022) · International Conference on Advanced Technologies for Signal and Image Processing · doi:10.1109/atsip55956.2022.9805959 · Cited by 11
Review Helpfulness Prediction Using Convolutional Neural Networks and Gated Recurrent Units
Basiri, M. E., & Habibi, S. (2020) · International Conference on Web Research (ICWR) · doi:10.1109/icwr49608.2020.9122297 · Cited by 11
“For example, some researchers studied the role of emotion [12], [13], others assessed the numerical and textual features of review [14], and some focused on statistical features such as review extremity and review length [15].”
BackgroundSelf-citationRevathi, K. L., & Satish, A. R. (2023) · Machine Learning Science and Technology · doi:10.1088/2632-2153/acc0d5 · Cited by 16
“Error DGFLP [16] 0.95 Latent space data fusion method [18] 0.92 Sentiment classification model [19] 0.98 AL-SSVAE [20] 0.86 SenticNet [21] 0.59 SITO [22] 0.14 The proposed NH-ResNeXt-RNF 0.039 Figure 14 depicts the proposed and existing outcome of a sentimental analysis classification system in…”
MethodInfluentialEmotion recognition with hybrid attentional multimodal fusion framework using cognitive augmentation
Kulkarni, S. V., Khot, S. S., & Angal, Y. (2025) · International Journal of Information Technology · doi:10.1007/s41870-025-02402-6 · Cited by 10
Multimodal Sentiment Analysis Using BiGRU and Attention-Based Hybrid Fusion Strategy
Liu, Z., Zhou, B., Meng, L., & Huang, G. (2023) · Intelligent Automation & Soft Computing · doi:10.32604/iasc.2023.038835 · Cited by 12
Methodological Review of Emotion Recognition for Social Media: A Sentiment Analysis Approach
Darokar, M. S., Raut, A. D., & Thakre, V. M. (2021) · International Conference on Computing, Communication and Green Engineering (CCGE) · doi:10.1109/ccge50943.2021.9776385 · Cited by 11
Efficient audio-visual emotion recognition approach
Alasiry, A., Al-Hussain, M. M., Alouane, M. T.-H., & Hadj-Alouane, N. B. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20572-6 · Cited by 13
“Also, in [10], these same modalities, along with a hybrid data fusion scheme, are the basis yet another for emotion recognition system.”
A Multimodal Affective Sensing Model for Constructing a Personality-Based Financial Advisor System
Lee, C.-H., Yang, H.-C., Su, X., & Tang, Y.-X. (2022) · Applied Sciences · doi:10.3390/app121910066 · Cited by 9
Sharma, M., Chen, S., Müller, P., Rekrut, M., & Krüger, A. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3577190.3614166 · Cited by 10
“It was successfully used in emotion recognition [10, 27] and multimedia event detection [21], to name a few.”
MethodPredicting the Helpfulness Score of Product Reviews Using an Evidential Score Fusion Method
Fouladfar, F., Dehkordi, M. N., & Basiri, M. E. (2020) · IEEE Access · doi:10.1109/access.2020.2988872 · Cited by 9
“D-S method is one of the most prominent score fusion methods which has been exploited in recent years for polarity detection [52], rating prediction [15], multimodal emotion recognition [53], and project risk assessment [54].”
MethodSelf-citationFeature selection enhancement and feature space visualization for speech-based emotion recognition
Kanwal, S., Asghar, S., & Ali, H. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1091 · Cited by 10
“Some studies reported emotion recognition based on speech features combined with other modalities, for example, text (Lee & Lee, 2007), facial expressions (Nemati et al., 2019; Bellegarda, 2013) body signals, and outward appearances (Yu & Tapus, 2019; Ozkul et al.”
BackgroundHe, X., Pan, Y., Guo, X., Xu, Z., & Yang, C. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3580779 · Cited by 10
“Common fusion strategies include feature-level fusion [7], [8], decision-level fusion [9], [10], and consistent regression fusion [11], [12].”
Multimodal fusion-powered English speaking robot
Pan, R. (2024) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2024.1478181 · Cited by 8
Bidirectional LSTM Deep Model for Online Doctor Reviews Polarity Detection
Basiri, M. E., Chegeni, R. S., Karimvand, A. N., & Nemati, S. (2020) · International Conference on Web Research (ICWR) · doi:10.1109/icwr49608.2020.9122289 · Cited by 8
“Also, feature engineering that is a vital step in traditional ML methods is a time-and labor-intensive task [13].”
BackgroundSelf-citationMOTIF: A text-aware and disentangled Mamba-based architecture for multimodal sentiment analysis
Hu, B., & Li, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114651 · Cited by 9
Semisupervised Deep Features of Time‐Frequency Maps for Multimodal Emotion Recognition
Zali-Vargahan, B., Charmin, A., Kalbkhani, H., & Barghandan, S. (2023) · International Journal of Intelligent Systems · doi:10.1155/2023/3608115 · Cited by 9
“A hybrid multimodal data fusion method was presented in [21] to fuse the audio and video signals from the DEAP dataset using a latent space linear map.”
MethodJony, R. I. (2023) · Queensland University of Technology · doi:10.5204/thesis.eprints.238058 · Cited by 9
Kothuri, S. R., & R, R. N. (2022) · Indian Journal of Computer Science and Engineering · doi:10.21817/indjcse/2022/v13i2/221302040 · Cited by 5
A Two-Step Framework to Recognize Emotion Using the Combinations of Adjacent Frequency Bands of EEG
Zhang, Z., & Zhang, L. (2023) · Applied Sciences · doi:10.3390/app13031954 · Cited by 7
Progress in Multimodal Affective Computing: From Machine Learning to Deep Learning
Chanchal, M., & Kumar, B. V. (2023) · EAI/Springer Innovations in Communication and Computing · doi:10.1007/978-3-031-20541-5_6 · Cited by 4
Srivastava, S., & Sharma, H. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16274-0 · Cited by 4
“The system can be hybrid either at the technique level or data level such as a fusion of multi-modalities [50].”
LUSTER: Link Prediction Utilizing Shared-Latent Space Representation in Multi-Layer Networks
Yang, R., Ali, M. A., Wang, H., Chen, J., & Wang, D. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714631 · Cited by 5
“For example, in multi-modal sentiment analysis [36], data from different modalities such as text, audio, and video can work together in a shared-latent space to more fully understand emotional expressions.”
Harata, S., Sakuma, T., & Kato, S. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-06388-6_18 · Cited by 4
Affective computing–driven virtual rehabilitation: a systematic survey
Zhao, L., Liu, D., Zhang, H., Yang, G., Zheng, D., Yan, J., & He, L. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00442-3 · Cited by 5
“For instance, Nemati et al. (Nemati et al. 2019) employed a linear mapping in the latent space to fuse audio and visual modalities at the feature level.”
Chien, G., Liu, J., Wang, J., & Chen, L. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115654 · Cited by 7
Analytical Review and Study on Emotion Recognition Strategies Using Multimodal Signals
Vala, J., & Jaliya, U. K. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-23092-9_21 · Cited by 3
Saisanthiya, D., & Supraja, P. (2023) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v17i16.42735 · Cited by 3
Multimodal Emotion Recognition Using Computer Vision: A Comprehensive Approach
Mohan, G. B., Sarrvesh, N., Hrishikeasan, N. P., Boopalamani, J., & Doss, S. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10725638 · Cited by 2
A Digital Twin-Based Approach for Emotion Recognition in Human-Robot Collaboration
Baratta, A., Longo, F., Mirabelli, G., Padovano, A., & Solina, V. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-38274-1_14 · Cited by 4
Sree, V. V. V. B., Mohan, G. B., Kumar, R. P., Rithani, M., & Pallavi, G. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10725013 · Cited by 2
“Several papers have proposed novel models that consider the temporal dynamics and interactions between modalities like text, audio, and video to obtain enhanced multimodal fusion representations [3, 4, 5 and 6].”
BackgroundZhang, T., Song, B., Zhang, Z., & Zhang, Y. (2024) · IET Image Processing · doi:10.1049/ipr2.13310 · Cited by 12
FGMFN: Fine-Grained Multiscale Cross-Modal Sentiment Analysis in Advertisements
Wang, H., Chen, P., & Du, X. (2025) · IEEE Access · doi:10.1109/access.2025.3571624 · Cited by 2
“This is particularly important in evaluating advertising effectiveness, as unimodal analysis often fails to capture the complex emotional responses of viewers to advertisements [6], [7], [8].”
Multimodal Raga Classification from Vocal Performances with Disentanglement and Contrastive Loss
Roychowdhury, S., & Rao, P. (2025) · Transactions of the International Society for Music Information Retrieval · doi:10.5334/tismir.221 · Cited by 2
Santhiya, P., Shanmugavadivel, K., Rajalakshmi, R., & Krishnamoorthy, N. (2025) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-025-00811-w · Cited by 2
“Nemati et al. [29] reported that this approach, tested on the DEAP dataset's video data, achieved an accuracy of 89.40%, outperforming decision-level and non-latent space fusion approaches, but reducing emotion detection effectiveness.”
InfluentialMohiddin, S. K., Sharmila, S., & Slimani, K. (2026) · Information systems engineering and management · doi:10.1007/978-3-032-14778-3_10 · Cited by 1
Beyond Words: Enhancing Desire, Emotion, and Sentiment Recognition with Non-Verbal Cues
Chen, W., Wang, T., Xue, F., Li, J., Liu, H., & Sha, Y. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792393 · Cited by 1
“(3) We provide comprehensive experiments and ablation studies on the MSED dataset, validating the e ff ectiveness and generalization of our proposed SyDES for multimodal desire understanding, emotion recognition, and sentiment analysis.”
Co-attention Guided Local-Global Feature Fusion for Aspect-Level Multimodal Sentiment Analysis
Cai, G., Wang, S., & Lv, G. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8429-9_30 · Cited by 1
Multimodal Visual Features on Rhizome Image Classification Using Support Vector Machines
Hamdani, S. K., Fahmi, H., Khudzaifah, M., Kusumastuti, A., Jauhari, M. N., & Sari, W. P. (2024) · doi:10.1109/icoict61617.2024.10698345 · Cited by 1
“By integrating these multimodal visual features, classification models can achieve a nuanced analysis of visual data, leading to enhanced accuracy and robustness in classification tasks [16].”
Cross-Subject Startle Detection for Pilots on the Flight Deck Using Physiological Signals
Basavaraju, G., Jungbluth, T., Rekrut, M., Daiber, F., & Krüger, A. (2024) · IEEE International Conference on Metrology for eXtended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE) · doi:10.1109/metroxraine62247.2024.10797063 · Cited by 1
“However future research should further explore innovative fusion strategies, such as Latent Space Fusion methods [14], to improve model reliability and generalization capabilities.”
Hegh, A. N., Adeyelu, A. A., Iorliam, A., & Otor, S. U. (2025) · FUDMA Journal of Sciences · doi:10.33003/fjs-2025-0905-3412 · Cited by 1
Chen, D., Li, H., & Zhu, Z. (2025) · ITM Web of Conferences · doi:10.1051/itmconf/20257802004 · Cited by 1
Vala, J., & Jaliya, U. K. (2023) · International Journal of Engineering Applied Sciences and Technology · doi:10.33564/ijeast.2023.v08i04.021 · Cited by 1
“Moreover, considering advanced genetic programming principles and transfer learning can also help in enhancing the accuracy and generalizability of emotion recognition systems [19,40].”
BackgroundHierachical Delta-Attention Method for Multimodal FusionPreprint
Panchal, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.10916 · Cited by 1
“And lastly, the mechanism of fusing the modalities itself is not clearly figured out, the papers focusing on late fusion [17, 11] fall short in making any cross-correlation between the effect of each modality on the classification, generally early fusion methods [10] have been proven superior, this…”
BackgroundBimodal Emotion Classification Using Deep Learning
Singh, A. K. (2020) · Arrow - TU Dublin (Technological University Dublin) · doi:10.21427/gpbz-2319
“While Nemati et al. (2019) presented a multi-modal approach that incorporates images and text explanations in real-world situations to enhance the efficiency of multi-modal classification.”
BackgroundMethodMachine and Deep Learning Technique for Depression Detection Using EEG Data
Tiwari, S., Pandey, K., Sharma, V. K., & Verma, B. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-8865-3_34
Evidence-Based Research on Multimodal Fusion Emotion Recognition
Huang, Z., & Liao, M. (2023) · Atlantis Highlights in Intelligent Systems/Atlantis highlights in intelligent systems · doi:10.2991/978-94-6463-200-2_61 · Cited by 1
Jaleel, M. U., Gupta, P., Gupta, P., Gupta, P. K., Bharti, S., Sehrawat, J., & Singh, A. (2023) · Journal Of Advanced Zoology · doi:10.17762/jaz.v44is-5.1088 · Cited by 1
REVIEW PAPER ON SENTIMENTAL ANALYSIS USING MACHINE LEARNING TECHNIQUES
(2023) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets47928
A Comprehensive Review of Multimodal Sentiment Analysis on Social Networks
Vinitha, V. S., & Jayanthi, R. (2024) · Algorithms for intelligent systems · doi:10.1007/978-981-97-0180-3_51
A multimodal health data fusion and deep analysis approach in smart campus systems
Fu, F., & Li, Z. (2025) · International Journal of Information and Communication Technology · doi:10.1504/ijict.2025.146674
Motif: A Text-Aware and Disentangled Mamba-Based Architecture for Multimodal Emotion RecognitionPreprint
Hu, B., & Li, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5351685
LGFN: A Dynamic Gating Framework for Lyrics-Audio Alignment in Music Emotion RecognitionPreprint
Wang, Y., Zang, M., Zhang, H., Shan, X., & Wang, X. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.11.669801
Revolutionizing Vision Perception with Multimodal Fusion Technologies
Batta, P., Chauhan, R., & Kaur, G. (2025) · Wiley eBooks · doi:10.1002/9781394272624.ch14
MAGI: Modality-Aligned Geometry-Aware Integration for Robust Multimodal Sentiment Analysis
Hu, B., & Li, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3349-7_46
Multilevel Feature Fusion and Attention Enhancement for Multimodal Sentiment Analysis
Zhang, L., He, T., Chen, B., & Shuan, C. (2025) · International Symposium on Computer and Information Processing Technology (ISCIPT) · doi:10.1109/iscipt67144.2025.11265256
“Hybrid fusion selects fusion timing based on model characteristics; for example, Nemati et al. [13] apply early fusion for audio and video, followed by late fusion with text to optimize performance.”
Multimodal Emotion Detection and Classification of Audio-Visual Using Deep Learning
Srivastava, M., Singh, P., & Sharma, M. K. (2025) · International Conference on Advanced Computing and Emerging Technologies (ACET) · doi:10.1109/acet67282.2025.11430197
“Salama et al. [16] developed a unique framework using 3D-CNN deep learning to derive spatiotemporal characteristics from EEG signals and human face video data.”
A Comprehensive Review on Enhancing Emotional Understanding Through Multimodal Fusion
Boby, M., Singh, Y., & Nikita (2026) · International Conference on Communication, Computing and Electronics Systems (ICCCES) · doi:10.1109/iccces62661.2026.11437288
Multimodal sentiment analysis: emerging innovations, core challenges, and future directions
Dhotay, M., Dharrao, M., Deokate, S. T., Bongale, A. M., & Dharrao, D. S. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01141-2 · Cited by 2
Hierarchical Structure Consistency Learning for Multimodal Emotion Recognition
Yang, B., Shi, Q., Zong, X., & Ye, M. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3694526
“Recent advancements in MER focus on two primary strategies: crossmodal attention-based methods [12], [13], [24], [25] and fusion-based techniques [15], [16], [18], [26], both of which have demonstrated significant improvements in recognition accuracy.”
TeFN: Text‐Centered Fusion Network With Multilevel Attention for Multimodal Sentiment Analysis
Chen, W., Liu, Y., Xiang, Y., & Hou, Y. (2026) · International Journal of Intelligent Systems · doi:10.1155/int/8485020
Metadata
- Authors
- Nemati, S., Rohani, R., Basiri, M. E., Abdar, M., Yen, N. Y., & Makarenkov, V.
- Affiliations
- Shahrekord University (IR)
- Université du Québec à Montréal (CA)
- University of Aizu (JP)
- Venue
- IEEE Access
- Published
- 2019
- Volume
- 7
- Pages
- 172948-172964
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2994718079
- Cited by
- 98
- References
- 88
- Citations by year
- 2026: 8 · 2025: 22 · 2024: 13 · 2023: 23 · 2022: 14 · 2021: 9 · 2020: 9
- FWCI
- 9.43
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Shahrekord University (97GRN1M44264)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal emotion recognition, Dempster-Shafer theory, evidential fusion, audiovisual integration, marginal Fisher analysis, cross-modal fusion, DEAP dataset, online comments, affective computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.