The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Multimodal emotion recognition model via hybrid model with improved feature level fusion on facial and EEG feature set
Abstract
This paper presents a multimodal emotion-recognition model using facial and EEG feature sets with improved feature-level fusion and a hybrid deep model. It targets better joint representation learning for emotion classification.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
International Journal of Innovative Research in Science, Engineering and Technology
(2014) · International Journal of Innovative Research in Science Engineering and Technology · doi:10.15680/ijirset · Cited by 1,090
CTNet: Conversational Transformer Network for Emotion Recognition
Lian, Z., Liu, B., & Tao, J. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3049898 · Cited by 296
Zhang, X., Liu, J., Shen, J., Li, S., Hou, K., Hu, B., Gao, J., Zhang, T., & Hu, B. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2987575 · Cited by 206
Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion
Cimtay, Y., Ekmekcioglu, E., & Caglar-Ozhan, S. (2020) · IEEE Access · doi:10.1109/access.2020.3023871 · Cited by 200
European signal processing conference
Jain, V. (1980) · IEEE Acoustics, Speech, and Signal Processing Newsletter · doi:10.1109/msp.1980.237652 · Cited by 189
Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems
Ayata, D., Yaslan, Y., & Kamasak, M. E. (2020) · Journal of Medical and Biological Engineering · doi:10.1007/s40846-019-00505-7 · Cited by 182 · cited 1×
“Because this strategy approaches the classification problem by extracting features from each modality independently, it cannot learn patterns that exist across several data modalities [14].”
Classification of Image using Convolutional Neural Network (CNN)
Hossain, M. A., & Sajib, M. S. A. (2019) · Global Journal of Computer Science and Technology · doi:10.34257/gjcstdvol19is2pg13 · Cited by 160
Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion
Siriwardhana, S., Kaluarachchi, T., Billinghurst, M., & Nanayakkara, S. (2020) · IEEE Access · doi:10.1109/access.2020.3026823 · Cited by 158
A multimodal hierarchical approach to speech emotion recognition from audio and text
Singh, P., Srivastava, R., Rana, K. P. S., & Kumar, V. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107316 · Cited by 123
Self-citationExpression-EEG Based Collaborative Multimodal Emotion Recognition Using Deep AutoEncoder
Zhang, H. (2020) · IEEE Access · doi:10.1109/access.2020.3021994 · Cited by 122
Multimodal Emotion Recognition Fusion Analysis Adapting BERT With Heterogeneous Feature Unification
Lee, S., Han, D. K., & Ko, H. (2021) · IEEE Access · doi:10.1109/access.2021.3092735 · Cited by 112
Automatic Emotion Recognition Using Temporal Multimodal Deep Learning
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2020) · IEEE Access · doi:10.1109/access.2020.3027026 · Cited by 101
A hybrid latent space data fusion method for multimodal emotion recognition
Nemati, S., Rohani, R., Basiri, M. E., Abdar, M., Yen, N. Y., & Makarenkov, V. (2019) · IEEE Access · doi:10.1109/access.2019.2955637 · Cited by 98
Stickel, C., Ebner, M., Steinbach-Nordmann, S., Searle, G., & Holzinger, A. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02707-9_70 · Cited by 92
A multimodal emotion recognition method based on facial expressions and electroencephalography
Tan, Y., Sun, Z., Duan, F., Solé-Casals, J., & Caiafa, C. F. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103029 · Cited by 88 · cited 2×
“[21] has proposed a multimodal emotion recognition approach by introducing an HRI system based on the low sense of disharmony.”
Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network
Zhang, Y., Cheng, C., & Zhang, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3049516 · Cited by 84
Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video
Wang, Z., Zhou, X., Wang, W., & Chen, L. (2020) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-019-01056-8 · Cited by 79
Christy, A., Vaithyasubramanian, S., Jesudoss, A. G., & Praveena, M. D. A. (2020) · International Journal of Speech Technology · doi:10.1007/s10772-020-09713-y · Cited by 78
Computational approaches for mining user’s opinions on the Web 2.0
Petz, G., Karpowicz, M., Fürschuß, H., Auinger, A., Stříteský, V., & Holzinger, A. (2014) · Information Processing & Management · doi:10.1016/j.ipm.2014.07.005 · Cited by 57
Video multimodal emotion recognition based on Bi-GRU and attention fusion
Huan, R., Shu, J., Bao, S.-L., Liang, R., Chen, P., & Chi, K. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10030-4 · Cited by 57
A Multimodal Emotion Recognition System Using Facial Landmark Analysis
Rahdari, F., Rashedi, E., & Eftekhari, M. (2018) · Iranian Journal of Science and Technology Transactions of Electrical Engineering · doi:10.1007/s40998-018-0142-9 · Cited by 41
Speech Expression Multimodal Emotion Recognition Based on Deep Belief Network
Liu, D., Chen, L., Wang, Z., & Diao, G. (2021) · Journal of Grid Computing · doi:10.1007/s10723-021-09564-0 · Cited by 38
Common spatial pattern-based feature extraction from the best time segment of BCI data
Aydemir, Ö. (2016) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.3906/elk-1502-162 · Cited by 34
Interactive Multimodal Attention Network for Emotion Recognition in Conversation
Ren, M., Huang, X., Shi, X., & Nie, W. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3078698 · Cited by 34
Facial Expression Recognition Model Depending on Optimized Support Vector Machine
Alhussan, A. A., Talaat, F. M., El‐kenawy, E. M., Abdelhamid, A. A., Ibrahim, A. F., Khafaga, D. S., & Alnaggar, M. (2023) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.039368 · Cited by 31
VLSI architecture for LGXP texture for face recognition
Kannan, P., & Kumari, R. S. S. (2014) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/ifs-1412366 · Cited by 23
Hybrid Model-Based Emotion Contextual Recognition for Cognitive Assistance Services
Ayari, N., Abdelkawy, H., Chibani, A., & Amirat, Y. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3013112 · Cited by 22
Emotion recognition with multimodal features and temporal models
Wang, S., Wang, W., Zhao, J., Chen, S., Jin, Q., Zhang, S., & Qin, Y. (2017) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3136755.3143016 · Cited by 14
Tripathi, M. K., & Shivendra, S. (2024) · Network Computation in Neural Systems · doi:10.1080/0954898x.2023.2299851 · Cited by 13
Self-citationNeutrosophic approach based intelligent system for automatic mango detection
Tripathi, M. K., & Shivendra, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17037-7 · Cited by 12
Self-citationK-Means Clustering-based Kernel Canonical Correlation Analysis for Multimodal Emotion Recognition
Chen, L., Wang, K., Wu, M., et al. (2020) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2020.12.2756 · Cited by 11
AGA-LSTM: An Optimized LSTM Neural Network Model Based on Adaptive Genetic Algorithm
Bai, C. (2020) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1570/1/012011 · Cited by 10
Wang, Y. (2019) · Personal and Ubiquitous Computing · doi:10.1007/s00779-018-01195-9 · Cited by 8
An Improved AAM Method for Extracting Human Facial Features
Zhou, T., Wu, X., Wu, T., et al. (2012) · Journal of Applied Mathematics · doi:10.1155/2012/643562 · Cited by 6
An Improved Feature Fusion for Speaker Recognition
Dai, M., Dai, G., Wu, Y., et al. (2019) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc.2019.00035 · Cited by 3
Research works printed but not linked: 5
- Aggarwal K, Mijwil MM, Al-Mistarehi AH, Alomari S, Gök M, Alaabdin AMZ, Abdulrhman SH (2022) Has the future started? The current growth of artificial intelligence, machine learning, and deep learning. Iraqi J Comput Sci Math 3(1):115–123
- Cimtay Y, Ekmekcioglu E, Caglar-Ozhan S (2020) Cross-subject multimodal emotion recognition based on hybrid fusion. IEEE Access 8:168865–168878. https://doi.org/10.1109/ACCESS.2020.3023871 · DOI · Paper page
- Guo H, Gao Z (2023) Multimodal sentiment recognition based on Bi-LSTM and fusion mechanism. Acad J Comput Inf Sci 6(6):127–132
- Ahad AR, Paul T, Shammi UA, Shammi UA (2018) A study on face detection using Viola-Jones algorithm for various backgrounds, angels and distances. IJBSCHS 23:27–36
- Duzenli T, Ozkurt N (2011) Comparison of wavelet-based feature extraction methods for speech/music discrimination. IU-JEEE 11:1355–1362
Cited by
Hybrid Fusion Paradigm in Advanced Process Monitoring: A Panoramic Review and Future Perspectives
Ali, H., Safdar, R., Liu, J., Manan, T. S. B. A., Hu, G., Rasool, M. H., Yao, Y., & Gao, F. (2025) · Industrial & Engineering Chemistry Research · doi:10.1021/acs.iecr.5c02759 · Cited by 63
Shaon, M. S. H., Karim, T., Shakil, M. S., & Hasan, M. Z. (2024) · Healthcare Analytics · doi:10.1016/j.health.2024.100353 · Cited by 43
An explainable Artificial Intelligence software system for predicting diabetes
Srinivasu, P. N., Ahmed, S., Hassaballah, M., & Almusallam, N. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e36112 · Cited by 42
Metaheuristics for pruning convolutional neural networks: A comparative study
Palakonda, V., Tursunboev, J., Kang, J., & Moon, S. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126326 · Cited by 32
Singh, S. K., & Patnaik, K. S. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.109173 · Cited by 24
Metadata
- Authors
- Singh, P., Tripathi, M. K., Patil, M. B., Shivendra, & Neelakantappa, M.
- Affiliations
- Galgotias University (IN)
- Vardhaman College of Engineering (IN)
- Corresponding author
- Tripathi, Mukesh Kumar
- Venue
- Multimedia Tools and Applications
- Published
- 2024-04-26
- Volume
- 84
- Issue
- 1
- Pages
- 1-36
- ISSN
- 1380-7501 (ISSN-L)
- 1573-7721
- OpenAlex
- W4395674629
- Cited by
- 38
- References
- 35
- Citations by year
- 2026: 7 · 2025: 25 · 2024: 6
- FWCI
- 15.57
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Multimedia Tools and Applications (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Keywords
- multimodal emotion recognition, EEG, facial expressions, feature fusion, wavelet features, common spatial pattern, texture features, gray level co-occurrence matrix, Viola-Jones face detection, LSTM, hybrid model
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.