The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion
Open accessAbstract
Multimodal emotion recognition has gained traction in affective computing research community to overcome the limitations posed by the processing a single form of data and to increase recognition robustness. In this study, a novel emotion recognition system is introduced, which is based on multiple modalities including facial expressions, galvanic skin response (GSR) and electroencephalogram (EEG). This method follows a hybrid fusion strategy and yields a maximum one-subject-out accuracy of 81.2% and a mean accuracy of 74.2% on our bespoke multimodal emotion dataset (LUMED-2) for 3 emotion classes: sad, neutral and happy. Similarly, our approach yields a maximum one-subject-out accuracy of 91.5% and a mean accuracy of 53.8% on the Database for Emotion Analysis using Physiological Signals (DEAP) for varying numbers of emotion classes, 4 in average, including angry, disgust, afraid, happy, neutral, sad and surprised. The presented model is particularly useful in determining the correct emotional state in the case of natural deceptive facial expressions. In terms of emotion recognition accuracy, this study is superior to, or on par with, the reference subject-independent multimodal emotion recognition studies introduced in the literature.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303 · cited 1×
“The face data used in this study for training our multimodal emotion recognition model is composed of well-known public face imagery datasets including Cohn-Kanade + face dataset [69], Radboud Faces Database [70], FacesDB [71] and AffectNet [72].”
MethodAffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338
The brain basis of emotion: a meta-analytic review
Lindquist, K. A., Wager, T. D., Kober, H., Bliss-Moreau, E., & Barrett, L. F. (2012) · Behavioral and Brain Sciences · doi:10.1017/s0140525x11000446 · Cited by 2,320 · cited 1×
“event or stimuli, the emotion is triggered first in the brain [76].”
BackgroundUniversals and cultural differences in the judgments of facial expressions of emotion.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.53.4.712 · Cited by 1,763
EmotionMeter: A Multimodal Framework for Recognizing Human Emotions
Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., & Cichocki, A. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2797176 · Cited by 1,276
Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852 · cited 1×
“In [29], a Multimodal Dictionary is presented to understand the relationship between the spoken words and facial gestures when introducing sentiment.”
BackgroundEnd-to-End Multimodal Emotion Recognition using Deep Neural Networks
Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2017) · arXiv · doi:10.48550/arxiv.1704.08619 · Cited by 752
The eNTERFACE’05 Audio-Visual Emotion Database
Martin, O., Kotsia, I., Macq, B., & Pitas, I. (2006) · 22nd International Conference on Data Engineering Workshops (ICDEW'06) · doi:10.1109/icdew.2006.145 · Cited by 615
Automatic prediction of frustration
Kapoor, A., Burleson, W., & Picard, R. W. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.02.003 · Cited by 592
Fusing audio, visual and textual clues for sentiment analysis from multimodal content
Poria, S., Cambria, E., Howard, N., Huang, G.-B., & Hussain, A. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.01.095 · Cited by 534 · cited 1×
“Multiple other recent studies under the topic of unimodal and multimodal sentiment or emotion analysis are reported in [30]–[41].”
BackgroundSentiment Analysis Is a Big Suitcase
Cambria, E., Poria, S., Gelbukh, A., & Thelwall, M. (2017) · IEEE Intelligent Systems · doi:10.1109/mis.2017.4531228 · Cited by 439
Multimodal emotion recognition using deep learning architectures
Ranganathan, H., Chakraborty, S., & Panchanathan, S. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477679 · Cited by 249
Multimodal Emotion Recognition Using Deep Neural Networks
Tang, H., Liu, W., Zheng, W.-L., & Lu, B.-L. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-70093-9_86 · Cited by 194
Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems
Ayata, D., Yaslan, Y., & Kamasak, M. E. (2020) · Journal of Medical and Biological Engineering · doi:10.1007/s40846-019-00505-7 · Cited by 182 · cited 1×
“The study in [66] uses a decision fusion on physiological data including photoplethysmography (PPG), a respiratory belt (RB) and fingertip temperature (FTT) and conducts one-”
MethodYin, Z., Wang, Y., Liu, L., Zhang, W., & Zhang, J. (2017) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2017.00019 · Cited by 155
Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition
Huang, Y., Yang, J., Liu, S., & Pan, J. (2019) · Future Internet · doi:10.3390/fi11050105 · Cited by 124
Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition
Huang, Y., Yang, J., Liao, P., & Pan, J. (2017) · Computational Intelligence and Neuroscience · doi:10.1155/2017/2107451 · Cited by 115 · cited 3×
“In [52], a decision level fusion was applied to outcomes from EEG and face modalities.”
BackgroundMethodA hybrid latent space data fusion method for multimodal emotion recognition
Nemati, S., Rohani, R., Basiri, M. E., Abdar, M., Yen, N. Y., & Makarenkov, V. (2019) · IEEE Access · doi:10.1109/access.2019.2955637 · Cited by 98 · cited 2×
“The literature about emotional data fusion involves three data fusion techniques, which are early fusion (feature fusion) [42], [43], late fusion (decision fusion) [44]–[46] and hybrid approaches [17], [47], [48].”
BackgroundMethodMetallinou, A., Lee, S., & Narayanan, S. (2010) · 2010 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2010.5494890 · Cited by 89
Research works printed but not linked: 86
- 10.1109/MIS.2017.23 · DOI · Paper page
- 10.1109/MIS.2017.57 · DOI
- 10.1109/MIS.2019.2904691 · DOI · Paper page
- 10.1109/MIS.2019.2954966 · DOI
- 10.1109/MIS.2020.2979203 · DOI
- 10.1109/MIS.2016.31 · DOI · Paper page
- 10.1109/MIS.2017.3711649 · DOI
- 10.1109/MIS.2018.033001419 · DOI
- 10.1109/MIS.2019.2899142 · DOI
- 10.1109/MIS.2017.22 · DOI
- 10.1109/MIS.2018.2882362 · DOI · Paper page
- hazer-rau, 2020, The uulmMAC database—A multimodal affective corpus for affective computing in human-computer interaction, SENSORS, 20, 2308, 10.3390/s20082308 · DOI
- 10.1109/TENCON.2016.7847950 · DOI
- masui, 2020, Continuous estimation of emotional change using multimodal affective responses, Proc IEEE/CVF Conf Comput Vis Pattern Recognit (CVPR) Workshops, 290
- 10.1109/ICIMIA.2017.7975659 · doi:10.1109/icimia.2017.7975659
- 10.1109/BDCloud-SocialCom-SustainCom.2016.29 · DOI
- 10.1109/MIS.2019.2925204 · DOI
- 10.1109/TNN.2010.2091281 · DOI · Paper page
- 10.1109/ICBDA.2017.8078794 · DOI
- 10.1109/TCYB.2016.2633306 · DOI · Paper page
- 10.1109/JSTSP.2017.2764438 · doi:10.1109/jstsp.2017.2764438 · Paper page
- 2020, Seeds Dataset
- tipping, 2001, Sparse Bayesian learning and the relevance vector machine, J Mach Learn Res, 1, 211
- 10.1145/2911996.2912059 · DOI
- 10.1109/CVPR.1993.1626170 · DOI
- 10.1145/2070481.2070487 · DOI
- 10.1007/3-540-45014-9_5 · doi:10.1007/3-540-45014-9_5
- reuderink, 2008, Decision-level fusion for audio-visual laughter detection, Machine Learning for Multimodal Interaction (MLMI), 5237, 137, 10.1007/978-3-540-85853-9_13 · DOI
- 10.1109/CIHLI.2013.6613272 · DOI
- 10.1109/ICCP.2018.8516437 · DOI
- 10.1080/10447318.2016.1159799 · DOI
- 10.1109/BIBM.2015.7359713 · DOI
- wang, 2020, A review of emotion sensing: Categorization models and algorithms, Multimedia Tools Appl, 1
- 10.1109/URAI.2018.8441795 · DOI
- 10.1109/CSICC.2009.5349653 · DOI
- 10.1109/ACCESS.2019.2914872 · DOI
- 10.1109/TechSym.2016.7872646 · DOI
- 10.1109/TAFFC.2015.2411280 · DOI
- 2020, LUMED-2 Dataset
- 2020, FacesDB
- 10.1080/02699930903485076 · DOI · Paper page
- 10.1016/j.jneumeth.2010.04.028 · DOI · Paper page
- 2020, DEAP Dataset
- 2020, Pretrained Deep Neural Networks
- 2020, IMOTIONS
- 2020, Top 14 EEG Hardware companies [Ranked]
- 10.1109/ACIIAsia.2018.8470379 · DOI
- kim, 2005, Integrating information from speech and physiological signals to achieve emotional sensitivity, Proc INTERSPEECH, 809, 10.21437/Interspeech.2005-380 · DOI
- 10.1109/CVPR.2016.374 · DOI · Paper page
- syeda, 2017, Cross-subject continuous emotion recognition using speech and body motion in dyadic interactions, Proc INTERSPEECH, 1731
- 10.1007/s10579-015-9300-0 · DOI
- 10.1186/s12911-017-0562-x · DOI
- szegedy, 2017, Inception-v4, inception-resnet and the impact of residual connections on learning, Proc AAAI, 4, 1
- 10.1515/sh-2016-0011 · DOI
- 2020, Empatica e4
- cimtay, 2020, Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition, SENSORS, 20, 2034, 10.3390/s20072034 · DOI · Paper page
- 10.1109/ICASSP.2013.6637858 · DOI
- zhang, 2019, Cross-subject EEG-based emotion recognition with deep domain confusion, Intelligent Robotics and Applications (ICIRA), 11740, 558, 10.1007/978-3-030-27526-6_49 · DOI
- 10.1088/1742-6596/705/1/012051 · DOI
- 10.1162/089976698300017467 · DOI · Paper page
- 10.1109/FSKD.2016.7603437 · DOI
- gong, 2012, Geodesic flow kernel for unsupervised domain adaptation, Proc IEEE Conf Comput Vis Pattern Recognit, 2066
- shi, 2012, Information-theoretical learning of discriminative clusters for unsupervised domain adaptation, Proc Intl Conf on Machine Learning (ICML) 2012, 1275
- 10.1109/ICCV.2013.368 · DOI · Paper page
- 10.3389/fncom.2019.00053 · DOI
- 10.1016/j.jksuci.2019.11.003 · DOI · Paper page
- 10.1109/JSEN.2018.2883497 · DOI · Paper page
- 10.3389/fnbot.2017.00019 · DOI · Paper page
- 10.1037/0022-3514.53.4.712 · DOI · Paper page
- 10.1109/T-AFFC.2012.36 · DOI
- fabien, 2015, AVEC 2015: The first affect recognition challenge bridging across audio, video, and physiological data, Proc of International Audio/Visual Emotion Challenge and Workshop, 3
- 10.1109/ACCESS.2019.2949741 · DOI · Paper page
- 10.1109/ACCESS.2017.2672722 · DOI
- 10.1109/ACCESS.2019.2928691 · DOI · Paper page
- 10.3390/e18090272 · DOI · Paper page
- osman, 2016, Multimodal Affect Recognition Current Approaches and Challenges
- 10.1016/j.knosys.2013.02.014 · DOI · Paper page
- arns, 2013, A decade of EEG theta/beta ratio research in ADHD a meta-analysis, 17, 374
- 10.1145/3332169.3333569 · DOI
- 10.1093/brain/awy251 · DOI · Paper page
- 10.1016/j.neuroimage.2006.09.024 · DOI · Paper page
- thejaswini, 2019, Analysis of EEG based emotion detection of DEAP and SEED-IV databases using SVM, International Journal of Recent Technology and Engineering, 8, 207
- 10.1371/journal.pone.0174949 · DOI · Paper page
- 10.3390/s130810273 · DOI · Paper page
- 10.1371/journal.pone.0214507 · DOI
- 10.1109/ICSPIS.2017.8311585 · DOI
Cited by
Khare, S. K., Blanes-Vidal, V., Nadimi, E. S., & Acharya, U. R. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102019 · Cited by 469
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network
Zhang, Y., Cheng, C., & Zhang, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3049516 · Cited by 84
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
Singh, P., Tripathi, M. K., Patil, M. B., Shivendra, & Neelakantappa, M. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19171-2 · Cited by 38
Metadata
- Authors
- Cimtay, Y., Ekmekcioglu, E., & Caglar-Ozhan, S.
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 168865-168878
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3084624816
- Cited by
- 200
- References
- 22
- FWCI
- 18.51
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- Institutional Links grant (352175665)
- Newton–Katip Celebi partnership between the U.K. and Turkey
- U.K. Department of Business, Energy and Industrial Strategy (BEIS) and The Scientific and Technological Research Council of Turkey (TUBITAK) and delivered by the British Council
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.