The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Multimodal Emotion Recognition in Response to Videos
Open accessAbstract
This paper presents a user-independent emotion recognition method with the goal of recovering affective tags for videos using electroencephalogram (EEG), pupillary response and gaze distance. We first selected 20 video clips with extrinsic emotional content from movies and online resources. Then, EEG responses and eye gaze data were recorded from 24 participants while watching emotional video clips. Ground truth was defined based on the median arousal and valence scores given to clips in a preliminary study using an online questionnaire. Based on the participants' responses, three classes for each dimension were defined. The arousal classes were calm, medium aroused, and activated and the valence classes were unpleasant, neutral, and pleasant. One of the three affective labels of either valence or arousal was determined by classification of bodily responses. A one-participant-out cross validation was employed to investigate the classification performance in a user-independent approach. The best classification accuracies of 68.5 percent for three labels of valence and 76.4 percent for three labels of arousal were obtained using a modality fusion strategy and a support vector machine. The results over a population of 24 participants demonstrate that user-independent emotion recognition can outperform individual self-reports for arousal assessments and do not underperform for valence assessments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
Development and validation of brief measures of positive and negative affect: The PANAS scales.
Watson, D., Clark, L. A., & Tellegen, A. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.6.1063 · Cited by 32,433
Welch, P. (1967) · IEEE Transactions on Audio and Electroacoustics · doi:10.1109/tau.1967.1161901 · Cited by 12,277
Libsvm : A library for support vector machines
Ferrari, V. (2008) · Medical Entomology and Zoology · Cited by 10,122
The Cognitive Structure of Emotions
Ortony, A., Clore, G. L., & Collins, A. (1988) · doi:10.1017/cbo9780511571299 · Cited by 6,208
(2005) · CTIT technical reports series · Cited by 5,307
What are emotions? And how can they be measured?
Scherer, K. R. (2005) · Social Science Information · doi:10.1177/0539018405058216 · Cited by 4,264
Combining Pattern Classifiers: Methods and Algorithms
Bagui, S. C. (2005) · Technometrics · doi:10.1198/tech.2005.s320 · Cited by 3,202
Looking at pictures: Affective, facial, visceral, and behavioral reactions
LANG, P. J., GREENWALD, M. K., BRADLEY, M. M., & HAMM, A. O. (1993) · Psychophysiology · doi:10.1111/j.1469-8986.1993.tb03352.x · Cited by 3,190
Combining Pattern Classifiers: Methods and Algorithms
Kuncheva, L. I. (2004) · Wiley eBooks · doi:10.1002/0471660264 · Cited by 3,005
Cited by
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Zheng, W.-L., & Lu, B.-L. (2015) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2015.2431497 · Cited by 2,595
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
EmotionMeter: A Multimodal Framework for Recognizing Human Emotions
Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., & Cichocki, A. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2797176 · Cited by 1,276
Emotions Recognition Using EEG Signals: A Survey
Alarcao, S. M., & Fonseca, M. J. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2714671 · Cited by 1,201
Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges
Gravina, R., Alinia, P., Ghasemzadeh, H., & Fortino, G. (2017) · Information Fusion · doi:10.1016/j.inffus.2016.09.005 · Cited by 899
Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806
A Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
Human Emotion Recognition: Review of Sensors and Methods
Dzedzickis, A., Kaklauskas, A., & Bucinskas, V. (2020) · Sensors · doi:10.3390/s20030592 · Cited by 582
A Review of Human Activity Recognition Methods
Vrigkas, M., Nikou, C., & Kakadiaris, I. A. (2015) · Frontiers in Robotics and AI · doi:10.3389/frobt.2015.00028 · Cited by 566
Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection
Soleymani, M., Asghari-Esfeden, S., Fu, Y., & Pantic, M. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2436926 · Cited by 552
Self-citationEEG Based Emotion Recognition: A Tutorial and Review
Li, X., Zhang, Y., Tiwari, P., Song, D., Hu, B., Yang, M., Zhao, Z., Kumar, N., & Marttinen, P. (2023) · ACM Computing Surveys · doi:10.1145/3524499 · Cited by 508
Emotion recognition using wireless signals
Zhao, M., Adib, F., & Katabi, D. (2016) · Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking · doi:10.1145/2973750.2973762 · Cited by 461
ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection
Hazarika, D., Poria, S., Mihalcea, R., Cambria, E., & Zimmermann, R. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1280 · Cited by 452
Liu, W., Qiu, J.-L., Zheng, W.-L., & Lu, B.-L. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3071170 · Cited by 452
Speech emotion recognition using convolutional and Recurrent Neural Networks
Lim, W., Jang, D., & Lee, T. (2016) · 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA) · doi:10.1109/apsipa.2016.7820699 · Cited by 405
EEG Based Emotion Recognition by Combining Functional Connectivity Network and Local Activations
Li, P., Liu, H., Si, Y., Li, C., Li, F., Zhu, X., Huang, X., Zeng, Y., Yao, D., Zhang, Y., et al. (2019) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2019.2897651 · Cited by 396
EEG-based emotion classification using deep belief networks
Zheng, W.-L., Zhu, J.-Y., Peng, Y., & Lu, B.-L. (2014) · 2014 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2014.6890166 · Cited by 367
Exploring EEG Features in Cross-Subject Emotion Recognition
Li, X., Song, D., Zhang, P., Zhang, Y., Hou, Y., & Hu, B. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00162 · Cited by 367
Real-Time Movie-Induced Discrete Emotion Recognition from EEG Signals
Liu, Y.-J., Yu, M., Zhao, G., Song, J., Ge, Y., & Shi, Y. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2660485 · Cited by 356
Verma, G. K., & Tiwary, U. S. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.11.007 · Cited by 354
Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346
Facial Emotion Recognition Using Transfer Learning in the Deep CNN
Akhand, M. A. H., Roy, S., Siddique, N., Kamal, M. A. S., & Shimamura, T. (2021) · Electronics · doi:10.3390/electronics10091036 · Cited by 333
Emotion Recognition from Multi-Channel EEG through Parallel Convolutional Recurrent Neural Network
Yang, Y., Wu, Q., Qiu, M., Wang, Y., & Chen, X. (2018) · 2018 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489331 · Cited by 325
Electroencephalograms during Mental Arithmetic Task Performance
Zyma, I., Tukaev, S., Seleznov, I., Kiyono, K., Popov, A., Chernykh, M., & Shpenkov, O. (2019) · Data · doi:10.3390/data4010014 · Cited by 311
Al-Nafjan, A., Hosny, M., Al-Ohali, Y., & Al-Wabil, A. (2017) · Applied Sciences · doi:10.3390/app7121239 · Cited by 300
Cross-Subject Emotion Recognition Using Flexible Analytic Wavelet Transform From EEG Signals
Gupta, V., Chopda, M. D., & Pachori, R. B. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2018.2883497 · Cited by 293
Emotion Recognition Using Eye-Tracking: Taxonomy, Review and Current Challenges
Lim, J. Z., Mountstephens, J., & Teo, J. (2020) · Sensors · doi:10.3390/s20082384 · Cited by 293
Learning deep physiological models of affect
Martinez, H. P., Bengio, Y., & Yannakakis, G. N. (2013) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2013.2247823 · Cited by 287
A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges
Mühl, C., Allison, B., Nijholt, A., & Chanel, G. (2014) · Brain-Computer Interfaces · doi:10.1080/2326263x.2014.912881 · Cited by 282
M3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i02.5492 · Cited by 253
A machine learning model for emotion recognition from physiological signals
Domínguez-Jiménez, J. A., Campo-Landines, K. C., Martínez-Santos, J. C., Delahoz, E. J., & Contreras-Ortiz, S. H. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2019.101646 · Cited by 232
Multimodal emotion recognition using EEG and eye tracking data
Zheng, W.-L., Dong, B.-N., & Lu, B.-L. (2014) · 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/embc.2014.6944757 · Cited by 230
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
Majid Mehmood, R., Du, R., & Lee, H. J. (2017) · IEEE Access · doi:10.1109/access.2017.2724555 · Cited by 213
Continuous Convolutional Neural Network with 3D Input for EEG-Based Emotion Recognition
Yang, Y., Wu, Q., Fu, Y., & Chen, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-04239-4_39 · Cited by 197
Shoumy, N. J., Ang, L.-M., Seng, K. P., Rahaman, D. M. M., & Zia, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102447 · Cited by 171
Automatic Emotion Recognition Using Temporal Multimodal Deep Learning
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2020) · IEEE Access · doi:10.1109/access.2020.3027026 · Cited by 101
A hybrid latent space data fusion method for multimodal emotion recognition
Nemati, S., Rohani, R., Basiri, M. E., Abdar, M., Yen, N. Y., & Makarenkov, V. (2019) · IEEE Access · doi:10.1109/access.2019.2955637 · Cited by 98
Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities
Savran, A., Cao, H., Nenkova, A., & Verma, R. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2362101 · Cited by 53
Horvat, M., Dobrinic, M., Novosel, M., & Jercic, P. (2018) · 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) · doi:10.23919/mipro.2018.8400184 · Cited by 49
Leveraging analytics to produce compelling and profitable film content
Behrens, R., Foutz, N. Z., Franklin, M., Funk, J., Gutierrez-Navratil, F., Hofmann, J., & Leibfried, U. (2021) · Journal of Cultural Economics · doi:10.1007/s10824-019-09372-1 · Cited by 39
Multi-Domain Encoding of Spatiotemporal Dynamics in EEG for Emotion Recognition
Cheng, C., Zhang, Y., Liu, L., Liu, W., & Feng, L. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3232497 · Cited by 29
Design of a hierarchy modular neural network and its application in multimodal emotion recognition
Li, W., Chu, M., & Qiao, J. (2019) · Soft Computing · doi:10.1007/s00500-018-03735-0 · Cited by 13
Metadata
- Authors
- Soleymani, M., Pantic, M., & Pun, T.
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2012-04
- Volume
- 3
- Issue
- 2
- Pages
- 211-223
- ISSN
- 1949-3045 (ISSN-L)
- OpenAlex
- W2164699598
- Cited by
- 722
- References
- 56
- FWCI
- 23.33
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.