The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Audio-Visual Emotion Recognition in Video Clips
Open accessAbstract
This paper presents a multimodal emotion recognition system, which is based on the analysis of audio and visual cues. From the audio channel, Mel-Frequency Cepstral Coefficients, Filter Bank Energies and prosodic features are extracted. For the visual part, two strategies are considered. First, facial landmarks' geometric relations, i.e., distances and angles, are computed. Second, we summarize each emotional video into a reduced set of key-frames, which are taught to visually discriminate between the emotions. In order to do so, a convolutional neural network is applied to key-frames summarizing videos. Finally, confidence outputs of all the classifiers from all the modalities are used to define a new feature space to be learned for final emotion label prediction, in a late fusion/stacking fashion. The experiments conducted on the SAVEE, eNTERFACE'05, and RML databases show significant performance improvements by our proposed system in comparison to current alternatives, defining the current state-of-the-art in all three databases.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795
Davis, S., & Mermelstein, P. (1980) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1980.1163420 · Cited by 5,465
Deep Expectation of Real and Apparent Age from a Single Image Without Facial Landmarks
Rothe, R., Timofte, R., & Van Gool, L. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-016-0940-3 · Cited by 896
Intonation and Its Uses: Melody in Grammar and Discourse
Bolinger, D. (1989) · Stanford University Press eBooks · doi:10.1515/9781503623125 · Cited by 686
Configural information in facial expression perception.
Calder, A. J., Young, A. W., Keane, J., & Dean, M. (2000) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037//0096-1523.26.2.527 · Cited by 650
The eNTERFACE’05 Audio-Visual Emotion Database
Martin, O., Kotsia, I., Macq, B., & Pitas, I. (2006) · 22nd International Conference on Data Engineering Workshops (ICDEW'06) · doi:10.1109/icdew.2006.145 · Cited by 615
Adaptive key frame extraction using unsupervised clustering
Zhuang, Y., Rui, Y., Huang, T. S., & Mehrotra, S. (2002) · Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269) · doi:10.1109/icip.1998.723655 · Cited by 505
EmoNets: Multimodal deep learning approaches for emotion recognition in video
Kahou, S. E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-Lewandowski, N., et al. (2016) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0195-2 · Cited by 412
Multi-Directional Multi-Level Dual-Cross Patterns for Robust Face Recognition
Ding, C., Choi, J., Tao, D., & Davis, L. S. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2462338 · Cited by 381
A real-time automated system for the recognition of human facial expressions
Anderson, K., & McOwan, P. W. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.854502 · Cited by 355
Intonation and Its Uses: Melody in Grammar and Discourse
Bolinger, D. L. (1989) · Cited by 328
Torous, J., Friedman, R., & Keshavan, M. (2014) · JMIR mHealth and uHealth · doi:10.2196/mhealth.2994 · Cited by 287
Machine Analysis of Facial Expressions
Pantic, M., & Stewart, M. (2007) · Face Recognition · doi:10.5772/4847 · Cited by 248
Multimodal information fusion application to human emotion recognition from face and speech
Mansoorizadeh, M., & Moghaddam Charkari, N. (2010) · Multimedia Tools and Applications · doi:10.1007/s11042-009-0344-2 · Cited by 166
Wang, Y., Guan, L., & Venetsanopoulos, A. N. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2012.2189550 · Cited by 155
Facial expression recognition in peripheral versus central vision: role of the eyes and the mouth
Calvo, M. G., Fernández-Martín, A., & Nummenmaa, L. (2013) · Psychological Research · doi:10.1007/s00426-013-0492-x · Cited by 149
Multimodal Emotion Recognition
Haq, S., & Jackson, P. J. B. (2010) · IGI Global eBooks · doi:10.4018/978-1-61520-919-4.ch017 · Cited by 148
Vocal-based emotion recognition using random forests and decision tree
Noroozi, F., Sapiński, T., Kamińska, D., & Anbarjafari, G. (2017) · International Journal of Speech Technology · doi:10.1007/s10772-017-9396-2 · Cited by 139
Self-citationAction Recognition by Learning Deep Multi-Granular Spatio-Temporal Video Representation
Li, Q., Qiu, Z., Yao, T., Mei, T., Rui, Y., & Luo, J. (2016) · International Conference on Multimedia Retrieval · doi:10.1145/2911996.2912001 · Cited by 133
Cooperative Learning and its Application to Emotion Recognition from Speech
Zhang, Z., Coutinho, E., Deng, J., & Schuller, B. W. (2014) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2014.2375558 · Cited by 124
Cloud-Assisted Speech and Face Recognition Framework for Health Monitoring
Hossain, M. S., & Muhammad, G. (2015) · Mobile Networks and Applications · doi:10.1007/s11036-015-0586-3 · Cited by 124
Combining Facial Dynamics With Appearance for Age Estimation
Dibeklioğlu, H., Alnajar, F., Salah, A. A., & Gevers, T. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2412377 · Cited by 110
Designing and Evaluating a Social Gaze-Control System for a Humanoid Robot
Zaraki, A., Mazzei, D., Giuliani, M., & Rossi, D. D. (2014) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2014.2303083 · Cited by 105
A Combined Rule-Based & Machine Learning Audio-Visual Emotion Recognition Approach
Seng, K. P., Ang, L.-M., & Ooi, C. S. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2588488 · Cited by 92
Choice of Mel filter bank in computing MFCC of a resampled speech
Kopparapu, S. K., & Laxminarayana, M. (2010) · International Conference on Information Science, Signal Processing and their Applications (ISSPA) · doi:10.1109/isspa.2010.5605491 · Cited by 87
Extending touch-less interaction on vision based wearable device
Lv, Z., Feng, S., Feng, L., & Li, H. (2015) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr.2015.7223380 · Cited by 86
Investigating keyframe selection methods in the novel domain of passively captured visual lifelogs
Doherty, A., Byrne, D., Smeaton, A. F., Jones, G. J. F., & Hughes, M. M. (2008) · ACM International Conference on Image and Video Retrieval (CIVR) · doi:10.1145/1386352.1386389 · Cited by 55
Efficiency of chosen speech descriptors in relation to emotion recognition
Kamińska, D., Sapiński, T., & Anbarjafari, G. (2017) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-017-0100-x · Cited by 53
Self-citationLOW-LEVEL FUSION OF AUDIO AND VIDEO FEATURE FOR MULTI-MODAL EMOTION RECOGNITION
Wimmer, M., Schuller, B. W., Arsić, D., Rigoll, G., & Radig, B. (2008) · International Conference on Computer Vision Theory and Applications · doi:10.5220/0001082801450151 · Cited by 52
A Study of Techniques for Facial Detection and Expression Classification
Hemalatha, G., & Sumathi, C. P. (2014) · International Journal of Computer Science & Engineering Survey · doi:10.5121/ijcses.2014.5203 · Cited by 46
Gharavian, D., Bejani, M., & Sheikhan, M. (2016) · Multimedia Tools and Applications · doi:10.1007/s11042-015-3180-6 · Cited by 45
Lüsi, I., Jacques, J. C. S., Gorbova, J., Baró, X., Escalera, S., Demirel, H., Allïk, J., Ozcinar, C., & Anbarjafari, G. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.102 · Cited by 45
Self-citationSurvey of the Facial Expression Recognition Research
Wu, T., Fu, S., & Yang, G. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31561-9_44 · Cited by 42
Audio Visual Emotion Recognition Based on Triple-Stream Dynamic Bayesian Network Models
Jiang, D., Cui, Y., Zhang, X., Fan, P., Ganzalez, I., & Hichem Sahli, B. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24600-5_64 · Cited by 40
Emotion Recognition for Affect Aware Video Games
Szwoch, M., & Szwoch, W. (2014) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-10662-5_28 · Cited by 39
Multi-modal Emotion Recognition Using Canonical Correlations and Acoustic Features
Gajšek, R., Štruc, V., & Mihelič, F. (2010) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2010.1005 · Cited by 35
Planet, S., & Iriondo, I. (2012) · Iberian Conference on Information Systems and Technologies (CISTI) · Cited by 28
A Novel Multimodal Emotion Recognition Approach for Affective Human Robot Interaction.
Burgos, F. A. C., Manso, L. J., & Núñez, P. (2015) · Cited by 28
Multimodal emotion recognition with automatic peak frame selection
Zhalehpour, S., Akhtar, Z., & Erdem, Ç. E. (2014) · International Conference on Innovations in Intelligent Systems and Applications (INISTA) · doi:10.1109/inista.2014.6873606 · Cited by 27
Towards multimodal emotion recognition: a new approach
Paleari, M., Huet, B., & Chellali, R. (2010) · ACM International Conference on Image and Video Retrieval (CIVR) · doi:10.1145/1816041.1816069 · Cited by 27
Kim, Y., & Provost, E. M. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654934 · Cited by 24
Multimodal recognition of emotions in car environments
Rothkrantz, L. J. M. (2009) · Cited by 23
Providing services using network-based humanoids in a home environment
Kim, K., Cha, Y., Park, J.-M., et al. (2011) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2011.6131135 · Cited by 21
Fusion of classifier predictions for audio-visual emotion recognition
Noroozi, F., Marjanovic, M., Njeguš, A., Escalera, S., & Anbarjafari, G. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7899608 · Cited by 21
Self-citationLearning collaborative decision-making parameters for multimodal emotion recognition
Huang, K., Lin, H.-Y. S., Chan, J.-C., & Kuo, Y.-H. (2013) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2013.6607472 · Cited by 17
Multimodal Emotion Recognition Using Deep Networks
Fadil, C., Alvarez, R., Martínez, C. E., Goddard, J., & Rufiner, H. L. (2015) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-319-13117-7_207 · Cited by 15
Emotion Recognition from Audio and Visual Data using F-score based Fusion
Gera, A., & Bhattacharya, A. (2014) · doi:10.1145/2567688.2567690 · Cited by 9
A Key Frame Selection-Based Facial Expression Recognition System
Guo, S., Pan, Y., Liao, Y.-C., Hsu, C. Y., Tsai, J.-S., & Chang, C. I. (2006) · International Conference on Innovative Computing, Information and Control · doi:10.1109/icicic.2006.383 · Cited by 7
Human Head Pose Estimation on SASE Database Using Random Hough Regression Forests
Lüsi, I., Escalera, S., & Anbarjafari, G. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-56687-0_12 · Cited by 4
Self-citation
Cited by
Survey on Emotional Body Gesture Recognition
Noroozi, F., Corneanu, C. A., Kaminska, D., Sapinski, T., Escalera, S., & Anbarjafari, G. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2874986 · Cited by 455
Self-citationFacial Emotion Recognition Using Transfer Learning in the Deep CNN
Akhand, M. A. H., Roy, S., Siddique, N., Kamal, M. A. S., & Shimamura, T. (2021) · Electronics · doi:10.3390/electronics10091036 · Cited by 333
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
Albanie, S., Nagrani, A., Vedaldi, A., & Zisserman, A. (2018) · Proceedings of the 26th ACM international conference on Multimedia · doi:10.1145/3240508.3240578 · Cited by 290
Automated emotion recognition: Current trends and future perspectives
Maithri, M., Raghavendra, U., Gudigar, A., Samanth, J., Barua, P. D., Murugappan, M., Chakole, Y., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106646 · Cited by 172
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142
Metadata
- Authors
- Noroozi, F., Marjanovic, M., Njegus, A., Escalera, S., & Anbarjafari, G.
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2019-01-01
- Volume
- 10
- Issue
- 1
- Pages
- 60-75
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W2624340939
- Cited by
- 266
- References
- 52
- FWCI
- 25.48
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.