Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals
Abstract
The purpose of this paper is twofold: (i) to investigate the emotion representation models and find out the possibility of a model with minimum number of continuous dimensions and (ii) to recognize and predict emotion from the measured physiological signals using multiresolution approach. The multimodal physiological signals are: Electroencephalogram (EEG) (32 channels) and peripheral (8 channels: Galvanic skin response (GSR), blood volume pressure, respiration pattern, skin temperature, electromyogram (EMG) and electrooculogram (EOG)) as given in the DEAP database. We have discussed the theories of emotion modeling based on i) basic emotions, ii) cognitive appraisal and physiological response approach and iii) the dimensional approach and proposed a three continuous dimensional representation model for emotions. The clustering experiment on the given valence, arousal and dominance values of various emotions has been done to validate the proposed model. A novel approach for multimodal fusion of information from a large number of channels to classify and predict emotions has also been proposed. Discrete Wavelet Transform, a classical transform for multiresolution analysis of signal has been used in this study. The experiments are performed to classify different emotions from four classifiers. The average accuracies are 81.45%, 74.37%, 57.74% and 75.94% for SVM, MLP, KNN and MMC classifiers respectively. The best accuracy is for 'Depressing' with 85.46% using SVM. The 32 EEG channels are considered as independent modes and features from each channel are considered with equal importance. May be some of the channel data are correlated but they may contain supplementary information. In comparison with the results given by others, the high accuracy of 85% with 13 emotions and 32 subjects from our proposed method clearly proves the potential of our multimodal fusion approach.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous studies found general recognition rates from 57.77% to 85.46% for different emotional states.
“Previous studies have provided evidence for general recognition rate from 57,77% to 85,46% for different emotional states”
From Survey on AI-Based Multimodal Methods for Emotion Detection (2019) · cites this paper as [66]
A larger set of analyzed signals and more recognized emotions provides better results.
“that more higher set of analyzed signals and more recognized emotions provides better results [64][65][66].”
From Survey on AI-Based Multimodal Methods for Emotion Detection (2019) · cites this paper as [66]
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · see citing works
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565 · see citing works
James, W. (1891) · Henry Holt and Company eBooks · doi:10.1037/11059-000 · Cited by 25,799 · see citing works
The expression of the emotions in man and animals.
Darwin, C. D. (1872) · J. Murray eBooks · doi:10.1037/10001-000 · Cited by 12,120 · see citing works
Fundamentals of speech recognition
Rabiner, L. R., & Juang, B.-H. F. (1993) · Cited by 7,680 · see citing works
Cognitive, social, and physiological determinants of emotional state.
Schachter, S., & Singer, J. (1962) · Psychological Review · doi:10.1037/h0046234 · Cited by 6,223 · see citing works
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,312 · see citing works
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093 · see citing works
A FRAMEWORK FOR REPRESENTING KNOWLEDGE
Minsky, M. (1988) · Elsevier eBooks · doi:10.1016/b978-1-4832-1446-7.50018-2 · Cited by 4,542 · see citing works
Patterns of cognitive appraisal in emotion.
Smith, C. A., & Ellsworth, P. C. (1985) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.48.4.813 · Cited by 4,068 · see citing works
Ekman, P. (1993) · American Psychologist · doi:10.1037/0003-066x.48.4.384 · Cited by 3,441 · see citing works
Handbook of Cognition and Emotion
Dalgleish, T., & Power, M. (1999) · Wiley eBooks · doi:10.1002/0470013494 · Cited by 2,789 · see citing works
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603 · see citing works
The psychology of computer vision
Winston, P. H. (1976) · Pattern Recognition · doi:10.1016/0031-3203(76)90020-0 · Cited by 2,576 · see citing works
Detecting Stress During Real-World Driving Tasks Using Physiological Sensors
Healey, J. A., & Picard, R. W. (2005) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2005.848368 · Cited by 2,207 · see citing works
Universals and cultural differences in the judgments of facial expressions of emotion.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.53.4.712 · Cited by 1,763 · see citing works
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664 · see citing works
Bolt, R. A. (1980) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/800250.807503 · Cited by 1,445 · see citing works
The world of emotions is not two-dimensional
Fontaine, J. R. J., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.02024.x · Cited by 1,179 · see citing works
Web-Based Database for Facial Expression Analysis
Pantić, M., Valstar, M., Rademaker, R. M., & Maat, L. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521424 · Cited by 1,099 · see citing works
Emotion recognition based on physiological changes in music listening
Kim, J., & André, E. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.26 · Cited by 1,079 · see citing works
Bolt, R. A. (1980) · ACM SIGGRAPH Computer Graphics · doi:10.1145/965105.807503 · Cited by 921 · see citing works
Early versus late fusion in semantic video analysis
Snoek, C. G. M., Worring, M., & Smeulders, A. W. M. (2005) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1101149.1101236 · Cited by 899 · see citing works
Evolutionary explanations of emotions
Nesse, R. M. (1990) · Human Nature · doi:10.1007/bf02733986 · Cited by 884 · see citing works
Facial expression recognition from video sequences: temporal and static modeling
Cohen, I., Sebe, N., Garg, A., Chen, L. S., & Huang, T. S. (2003) · Computer Vision and Image Understanding · doi:10.1016/s1077-3142(03)00081-x · Cited by 874 · see citing works
Multimodal Emotion Recognition in Response to Videos
Soleymani, M., Pantić, M., & Pun, T. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.37 · Cited by 725 · see citing works
EMOTIONS: A COGNITIVE–PHENOMENOLOGICAL ANALYSIS
Lazarus, R. S., Kanner, A. D., & Folkman, S. (1980) · Elsevier eBooks · doi:10.1016/b978-0-12-558701-3.50014-4 · Cited by 621 · see citing works
The logic of typed feature structures
Carpenter, B. (1992) · Cited by 600 · see citing works
A method of combining multiple experts for the recognition of unconstrained handwritten numerals
Huang, Y., & Suen, C. Y. (1995) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.368145 · Cited by 566 · see citing works
Language and the Politics of Emotion
Lutz, C. A. (1992) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-199203000-00015 · Cited by 555 · see citing works
A psychoevolutionary theory of emotions
Plutchik, R. (1982) · Social Science Information · doi:10.1177/053901882021004003 · Cited by 504 · see citing works
Inferring psychological significance from physiological signals.
Cacioppo, J. T., & Tassinary, L. G. (1990) · American Psychologist · doi:10.1037//0003-066x.45.1.16 · Cited by 486 · see citing works
Oviatt, S. L. (2007) · Human factors and ergonomics · doi:10.1201/9781410615862.ch21 · Cited by 485 · see citing works
Inferring psychological significance from physiological signals.
Cacioppo, J. T., & Tassinary, L. G. (1990) · American Psychologist · doi:10.1037/0003-066x.45.1.16 · Cited by 481 · see citing works
Circumplex models of personality and emotions.
(1997) · American Psychological Association eBooks · doi:10.1037/10261-000 · Cited by 461 · see citing works
Nicolaou, M. A., Gunes, H., & Pantic, M. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.9 · Cited by 450 · see citing works
The Vera am Mittag German audio-visual emotional speech database
Grimm, M., Kroschel, K., & Narayanan, S. S. (2008) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2008.4607572 · Cited by 413 · see citing works
Neural Correlates of Processing Valence and Arousal in Affective Words
Lewis, P., Critchley, H., Rotshtein, P., & Dolan, R. (2006) · Cerebral Cortex · doi:10.1093/cercor/bhk024 · Cited by 391 · see citing works
Douglas‐Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O. M., McRorie, M., Martin, J., Devillers, L. Y., Abrilian, S., Batliner, A. M., et al. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_43 · Cited by 323 · see citing works
Dynamic Bayesian Networks for Audio-Visual Speech Recognition
Nefian, A., Liang, L., Pi, X., Liu, X., & Murphy, K. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206083 · Cited by 320 · see citing works
Juslin, P. N., & Scherer, K. R. (2005) · doi:10.1093/oso/9780198529613.003.0003 · Cited by 318 · see citing works
Grandjean, D., Sander, D., & Scherer, K. R. (2008) · Consciousness and Cognition · doi:10.1016/j.concog.2008.03.019 · Cited by 302 · see citing works
The SEMAINE corpus of emotionally coloured character interactions
McKeown, G. J., Valstar, M., Cowie, R., & Pantić, M. (2010) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2010.5583006 · Cited by 282 · see citing works
Multimodal approaches for emotion recognition: a survey
Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2005) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.600746 · Cited by 154 · see citing works
A graphical model for audiovisual object tracking
Beal, M. J., Jojić, N., & Attias, H. T. (2003) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2003.1206512 · Cited by 117 · see citing works
Blattner, M. M., & Glinert, E. P. (1996) · IEEE Multimedia · doi:10.1109/93.556457 · Cited by 105 · see citing works
A Comparative Evaluation of Fusion Strategies for Multimodal Biometric Verification
Fierrez-Aguilar, J., Ortega-García, J., Garcia-Romero, D., et al. (2003) · Lecture notes in computer science · doi:10.1007/3-540-44887-x_96 · Cited by 103 · see citing works
Machine Learning in Computer Vision
Sebe, N., Cohen, I., Garg, A., & Huang, T. S. (2005) · Springer eBooks · doi:10.1007/1-4020-3275-7 · Cited by 101 · see citing works
Real-Time Vision for Human-Computer Interaction
Kisačanin, B., Pavlović, V., & Huang, T. S. (2005) · Springer eBooks · doi:10.1007/0-387-27890-7 · Cited by 98 · see citing works
Interdependencies among Voice Source Parameters in Emotional Speech
Sundberg, J., Patel, S. A., Björkner, E., & Scherer, K. R. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.14 · Cited by 90 · see citing works
How do correlation and variance of base-experts affect fusion in biometric authentication tasks?
Poh, N., & Samy Bengio, A. (2005) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2005.857006 · Cited by 77 · see citing works
Information assimilation framework for event detection in multimedia surveillance systems
Atrey, P. K., Kankanhalli, M., & Jain, R. (2006) · Multimedia Systems · doi:10.1007/s00530-006-0063-8 · Cited by 71 · see citing works
Confidence measures for multimodal identity verification
Bengio, S., Marcel, C., Marcel, S., et al. (2002) · Information Fusion · doi:10.1016/s1566-2535(02)00089-1 · Cited by 70 · see citing works
Recognizing Affect from Linguistic Information in 3D Continuous Space
Schuller, B. W. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.17 · Cited by 69 · see citing works
Fusion of AV features and external information sources for event detection in team sports video
Xu, H., & Chua, T. (2006) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/1126004.1126007 · Cited by 54 · see citing works
A New Fractional Random Wavelet Transform for Fingerprint Security
Bhatnagar, G., Wu, Q. M. J., & Raman, B. (2011) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/tsmca.2011.2147307 · Cited by 51 · see citing works
Multimodal fusion using learned text concepts for image categorization
Zhu, Q., Yeh, M.-C., & Cheng, K.-T. T. (2006) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1180639.1180698 · Cited by 48 · see citing works
Context based multimodal fusion
Pfleger, N. (2004) · International Conference on Multimodal Interfaces · doi:10.1145/1027933.1027977 · Cited by 47 · see citing works
Audio-Visual Speech Synchrony Measure for Talking-Face Identity Verification
Bredin, H., & Chollet, G. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366215 · Cited by 43 · see citing works
Information-theoretic semantic multimedia indexing
Magalhães, J., & Rüger, S. (2007) · ACM International Conference on Image and Video Retrieval (CIVR) · doi:10.1145/1282280.1282368 · Cited by 43 · see citing works
Tracking Humans using Multi-modal Fusion
Zou, X., & Bhanu, B. (2006) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvpr.2005.545 · Cited by 42 · see citing works
Continuous audio–visual digit recognition using N-best decision fusion
Meyer, G. F., Mulligan, J. B., & Wuerger, S. M. (2003) · Information Fusion · doi:10.1016/j.inffus.2003.07.001 · Cited by 31 · see citing works
Enhancing Universal Access – EEG Based Learnability Assessment
Stickel, C., Fink, J., & Holzinger, A. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73283-9_88 · Cited by 31 · see citing works
COLOR IMAGE SEGMENTATION USING THE DEMPSTER-SHAFER THEORY OF EVIDENCE FOR THE FUSION OF TEXTURE
Mena, J. B. (2003) · Cited by 30 · see citing works
Adaptive multimodal fusion by uncertainty compensation
Pitsikalis, V., Katsamanis, A., Papandreou, G., et al. (2006) · Interspeech (USB) · doi:10.21437/interspeech.2006-616 · Cited by 26 · see citing works
Exploiting the dependencies in information fusion
Pan, H., Liang, Z., & Huang, T. S. (2003) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1999.784713 · Cited by 21 · see citing works
Video Classification Based On Low-Level Feature Fusion Model
Guironnet, M., Pellerin, D., & Rombaut, M. (2005) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.5281/zenodo.39005 · Cited by 19 · see citing works
Multi-sensory and Multi-modal Fusion for Sentient Computing
Town, C. P. (2006) · International Journal of Computer Vision · doi:10.1007/s11263-006-7834-8 · Cited by 19 · see citing works
Multimodal Human-Computer Interaction
Turk, M. (2005) · Springer eBooks · doi:10.1007/0-387-27890-7_16 · Cited by 16 · see citing works
Layered Dynamic Mixture Model for Pattern Discovery in Asynchronous Multi-modal Streams
Xie, L., Kennedy, L., Chang, S., et al. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1415589 · Cited by 13 · see citing works
An image recognition method based on multiple BP neural networks fusion
Ni, J., Ma, X., Xu, L., et al. (2005) · IEEE International Conference on Information Acquisition · doi:10.1109/icia.2004.1373380 · Cited by 13 · see citing works
Oliveira, A. M., Teixeira, M., da Fonseca, I. B., et al. (2006) · Proceedings of Fechner Day · Cited by 12 · see citing works
DS theory based fingerprint classifier fusion with update rule to minimize training time
Singh, R., Vatsa, M., Noore, A., et al. (2006) · IEICE Electronics Express · doi:10.1587/elex.3.429 · Cited by 9 · see citing works
From multi-sensor surveillance towards smart interactive spaces
Gandetto, M., Marchesooti, L., Sciutto, S., et al. (2003) · doi:10.1109/icme.2003.1220999 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806 · see citing works
Recognition of emotions using multimodal physiological signals and an ensemble deep learning model
Yin, Z., Zhao, M., Wang, Y., Yang, J., & Zhang, J. (2017) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2016.12.005 · Cited by 416 · see citing works
Emotion Recognition Using Multimodal Deep Learning
Liu, W., Zheng, W.-L., & Lu, B.-L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46672-9_58 · Cited by 301 · see citing works
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works
Survey on AI-Based Multimodal Methods for Emotion Detection
Marechal, C., Mikołajewski, D., Tyburek, K., Prokopowicz, P., Bougueroua, L., Ancourt, C., & Węgrzyn-Wolska, K. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-16272-6_11 · Cited by 137 · see citing works
Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition
Huang, Y., Yang, J., Liu, S., & Pan, J. (2019) · Future Internet · doi:10.3390/fi11050105 · Cited by 124 · see citing works
Wu, D., Zhang, J., & Zhao, Q. (2020) · IEEE Access · doi:10.1109/access.2020.3010311 · Cited by 51 · see citing works
Goshvarpour, A., Abbasi, A., & Goshvarpour, A. (2017) · Australasian Physical & Engineering Sciences in Medicine · doi:10.1007/s13246-017-0571-1 · Cited by 46 · see citing works
Design of a hierarchy modular neural network and its application in multimodal emotion recognition
Li, W., Chu, M., & Qiao, J. (2019) · Soft Computing · doi:10.1007/s00500-018-03735-0 · Cited by 13 · see citing works
Chen, Y., Bai, Z., Cheng, M., Liu, Y., Zhao, X., & Song, Y. (2023) · 2023 42nd Chinese Control Conference (CCC) · doi:10.23919/ccc58697.2023.10239741 · Cited by 3 · see citing works
- Authors
- Verma, G. K., & Tiwary, U. S.
- Affiliations
- Indian Institute of Information Technology Allahabad (IN)
- Corresponding author
- Verma, Gyanendra K.
- Venue
- NeuroImage
- Publisher
- Elsevier BV
- Published
- 2014-11
- Volume
- 102
- Pages
- 162-172
- ISSN
- 1053-8119 (ISSN-L)
- OpenAlex
- W1963753144
- Cited by
- 354
- References
- 75
- Citations by year
- 2026: 12 · 2025: 35 · 2024: 31 · 2023: 39 · 2022: 48 · 2021: 27 · 2020: 38 · 2019: 38 · 2018: 34 · 2017: 26 · 2016: 19 · 2015: 6
- FWCI
- 1.38
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- NeuroImage (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- emotion recognition, multimodal fusion, physiological signals, multiresolution analysis, discrete wavelet transform, EEG, DEAP database, emotion representation, appraisal theory, basic emotions, peripheral physiological signals, emotion classification
- MeSH terms
- Emotions, Humans, Models, Neurological, Electrophysiological Phenomena, Wavelet Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
